Natural Jammers: When Non-Robust Features Become Antagonistic

Project page: https://jiamingzhang.netlify.app/naturaljammers

Introduction

Deep neural networks (DNNs) achieve remarkable success on visual tasks, yet their reasoning differs fundamentally from human perception. Rather than relying solely on semantic content, DNNs are highly sensitive to non-robust features—subtle, high-frequency patterns that predict labels but remain imperceptible to humans Ilyas et al. [ [13], Gyevnar and Kasirzadeh [2025], Liu et al. [ [21]. The prevailing view treats these features as opportunistic shortcuts: brittle patterns that standard models exploit as a crutch to reach correct decisions Geirhos et al. [ [8], Kauffmann et al. [ [15], Eriksson et al. [ [4]. This framing assumes non-robust signals and semantic labels work synergistically.

We challenge this synergistic view by uncovering natural feature antagonism. As Figure 1 illustrates, non-robust signals do not always aid classification—for a substantial fraction of natural images, they actively mislead standard inference. We term these signals Natural Jammers. This creates an empirical paradox: although the robustness–accuracy trade-off suggests standard models ($f_{\mathrm{std}}$) should upper-bound clean performance Tsipras et al. [ [35], Suzuki et al. [ [32], Zhang et al. [ [40], Kafali et al. [ [14], Li and Li [ [20], we identify a distinct Jammed Set ($\mathcal{J}$) where robustly trained models ($f_{\mathrm{rob}}$) significantly outperform their standard counterparts on unmodified images. The standard model’s reliance on non-robust features becomes a liability—it is misled by brittle cues that contradict the label, while the robust model often succeeds because it is less sensitive to such cues. We employ $f_{\mathrm{rob}}$ as a diagnostic lens to disentangle robust semantic cues from non-robust correlations.

To explain this interference, we propose the Spectral-Spatial Locking mechanism through controlled experiments. Natural jammers arise from high-frequency patterns, but their disruptive influence depends critically on alignment with specific spatial configurations. This phase-locked coupling explains a striking duality: while these features mislead standard inference, they are also fragile. Breaking the lock—through spectral filtering (the Cleansing Paradox, § 3.1) or minor geometric shifts (the Sensitivity Inversion, § 3.2)—renders the jammer inert, allowing the model’s discriminative capacity to resurface.

This fragility raises a puzzle: if individual jammers are so easily disrupted, why does the model not achieve robust generalization? Our analysis reveals the Substitution Paradox: a $2\%$ edge crop removes about one third of the images in the Jammed Set, yet its total size remains nearly unchanged. Natural non-robustness is not a defect of particular images but a persistent property of the model–data geometry. When one jammer’s spatial-spectral lock breaks, new jammers emerge elsewhere, keeping the overall vulnerability level nearly constant. Non-robustness is redistributed rather than eliminated by simple augmentations.

Our contributions are:

• Feature Antagonism: We formalize the Jammed Set ($\mathcal{J}$), showing that a significant portion of standard model errors stem from non-robust signals that contradict semantic labels.

• Spectral-Spatial Locking: We provide causal evidence that non-robust features require precise alignment between high-frequency content and spatial structure to remain effective.

• Substitution Paradox: We demonstrate that jamming is a stable, systemic property—individual non-robust features are fragile and interchangeable, following a redistribution pattern under geometric transformations.

Landscape of Natural Non-Robustness

To move beyond a binary view of robust vs. non-robust features, we formalize a taxonomy based on the latent structure of the data distribution $\mathcal{D}$.

The interaction between these manifolds determines whether a model’s reliance on non-robust signals yields an opportunistic shortcut or a predictive failure. We utilize a standard classifier $f_{\mathrm{std}}$ and an adversarially trained counterpart $f_{\mathrm{rob}}$ to analyze these divergent behaviors.

Formal Taxonomy: The Quad-Set Partition

Given a sample $(x, y) \in\mathcal{D}$ and a chosen reference pair $(f_{\mathrm{std}}, f_{\mathrm{rob}})$, we partition samples into four disjoint sets based on their predictions on the original clean input $x$:

• Common Set ($\mathcal{C}$): $\{x \mid f_{\mathrm{std}}(x) = y \land f_{\mathrm{rob}}(x) = y\}$. Samples where features in $\mathcal{Z}_r$ and $\mathcal{Z}_{nr}$ are synergistic.

• Spurious Set ($\mathcal{S}$): $\{x \mid f_{\mathrm{std}}(x) = y \land f_{\mathrm{rob}}(x) \ne y\}$. The classic shortcut regime where $f_{\mathrm{std}}$ succeeds via brittle patterns in $\mathcal{Z}_{nr}$.

• Jammed Set ($\mathcal{J}$): $\{x \mid f_{\mathrm{std}}(x) \ne y \land f_{\mathrm{rob}}(x) = y\}$. Samples where $f_{\mathrm{std}}$ is misguided by antagonistic signals in $\mathcal{Z}_{nr}$.

• Uninformative Set ($\mathcal{G}$): $\{x \mid f_{\mathrm{std}}(x) \ne y \land f_{\mathrm{rob}}(x) \ne y\}$. Samples where signals in both manifolds are insufficient for classification.

While $\mathcal{S}$ represents the well-documented reliance on human-irrelevant shortcuts, $\mathcal{J}$ exposes a more pathological interaction where non-robust features actively contradict the ground-truth logic:

Empirical Prevalence and Stability

We conduct a representative audit on the ImageNet validation set Russakovsky et al. [ [27] ($N = 50{,}000$) to evaluate the prevalence of these sets across different architectures. The membership of the set is defined on the original inputs and, therefore, on the unmodified images $\operatorname{Acc}_{f_{\mathrm{std}}}(\mathcal{S}) = 100\%$ and $\operatorname{Acc}_{f_{\mathrm{std}}}(\mathcal{J}) = 0\%$ by construction (and analogously for $f_{\mathrm{rob}}$). As shown in Table 1, the Jammed Set $\mathcal{J}$ consistently accounts for a significant portion of the error manifold.

Model	$\mathcal{J}$ (Jammed)	$\mathcal{S}$ (Spurious)	$\mathcal{C}$ (Common)	$\mathcal{G}$ (Uninformative)
ResNet-18 ($\ell_2$)	2,448 (4.90%)	6,168 (12.34%)	28,712 (57.42%)	12,672 (25.34%)
ResNet-50 ($\ell_2$)	2,140 (4.28%)	5,002 (10.00%)	33,071 (66.14%)	9,787 (19.57%)
WRN-50-2 ($\ell_2$)	2,167 (4.33%)	4,698 (9.40%)	34,543 (69.09%)	8,592 (17.18%)
ResNet-18 ($\ell_\infty$)	2,502 (5.00%)	5,659 (11.32%)	29,221 (58.44%)	12,618 (25.24%)
ResNet-50 ($\ell_\infty$)	2,252 (4.50%)	4,306 (8.61%)	33,767 (67.53%)	9,675 (19.35%)
WRN-50-2 ($\ell_\infty$)	2,232 (4.46%)	4,145 (8.29%)	35,096 (70.19%)	8,527 (17.05%)

The consistency of these proportions (4.3%–5.0% for $\mathcal{J}$) across architectures suggests that natural jamming is not confined to a single evaluated model. However, this statistical stability does not necessarily imply that the “Jammed” status is an intrinsic property of the image. If the effect stems from universal data pathologies, the samples in $\mathcal{J}$ should overlap significantly across different models. Conversely, if it is a manifestation of specific inductive biases, these sets should diverge. We investigate this question by dissecting the spectral and spatial dependencies of these features in the next section.

Spectral-Spatial Locking: The Core Evidence

The divergent behaviors of $f_{\mathrm{std}}$ and $f_{\mathrm{rob}}$ on the Jammed and Spurious sets suggest that these models prioritize different structural components within the same image. In this section, we conduct a series of diagnostic experiments to isolate whether these non-robust features are defined by their frequency content, their spatial location, or a combination of both. Our results lead to the discovery of a Spectral-Spatial Locking mechanism.

Evaluation convention. All sets $\mathcal{C}$, $\mathcal{S}$, $\mathcal{J}$, $\mathcal{G}$ are defined using predictions on the original clean inputs. We then apply transformations and evaluate predictions on the same fixed subset of samples, without re-computing set membership after transformation. Unless stated otherwise, all experiments in this section use the primary lens (standard vs. robust ResNet-50, $\ell_2$, $\epsilon= 1$; checkpoints from Salman et al. [ [29]).

The Cleansing Paradox: Spectral Sensitivity

We begin by testing whether the interference in the Jammed Set is an intrinsic property of high-frequency (HF) patterns. We apply a low-pass filter (LPF) to progressively remove HF components and observe the accuracy of the standard model $f_{\mathrm{std}}$.

Protocol. For each image $x$, we compute its 2D Fourier transform $\mathcal{F}(x)$, apply a circular hard-cutoff mask $M_r$ with varying radius $r \in[10,112]$ (on a $224 \times224$ spectrum), and reconstruct the filtered image $\tilde{x}_r = \mathcal{F}^{-1}(M_r \cdot\mathcal{F}(x))$. A smaller $r$ corresponds to more aggressive high-frequency removal.

Results. As shown in Figure 2a, removing HF content yields opposite outcomes for the two sets: Spurious Set ($\mathcal{S}$): Accuracy collapses from 100.0% to 23.7% at $r = 30$. This confirms that these predictions rely entirely on high-frequency shortcuts. Jammed Set ($\mathcal{J}$): Accuracy recovers from 0.0% to 31.8% at $r = 50$. Common Set ($\mathcal{C}$): Accuracy degrades gracefully from 100.0% to 85.1% at $r = 50$, demonstrating that robust semantic features are largely preserved in the low-frequency skeleton. These findings are diagnostic evidence of the spectral fragility of non-robust features: their predictive utility is tied to specific frequency supports (HF patterns).

The Sensitivity Inversion: Spatial Localization

To determine where these jammers are located, we examine model sensitivity to spatial transformations.

Protocol. We apply a center-crop zoom $T_{\alpha}$, where the image is cropped to $\alpha\times224$ pixels and resized back to $224 \times224$ with bilinear interpolation. We sweep the ratio $\alpha$ from 1.0 (original) to 0.5 (2× zoom).

Results. Figure 2(b) reveals a distinct Sensitivity Inversion: as we zoom in, accuracy on Jammed Set ($\mathcal{J}$) increases while accuracy on Spurious Set ($\mathcal{S}$) decreases. Since zooming removes peripheral regions, this pattern is consistent with the hypothesis that many interfering cues are spatially localized near image boundaries or object context, where they can conflict with central semantic evidence. The accuracy on Common Set ($\mathcal{C}$) remains relatively stable, gracefully decreasing from 100% to 79.9%. Robust semantic features exhibit geometric invariance. This sensitivity highlights the spatial fragility of the underlying non-robust signals. This recovery is not specific to the resampling method: it appears with all six interpolation kernels we tested and when a 2% border is painted over instead of cropped.

Frequency Grafting: A Causality Test

The preceding experiments suggest a strong coupling between spectral content and spatial structure. To provide causal evidence, we test whether the interference in $\mathcal{J}$ is an intrinsic HF signal (analogous to a universal adversarial perturbation) or if its efficacy is strictly dependent on the host image’s geometry. We perform a frequency grafting experiment that transplants the HF components of a Jammed image onto the LF skeleton of a Common image.

Protocol. Let $x_j \in\mathcal{J}$ be a Jammed image and $x_c \in\mathcal{C}$ be a Common image. We construct a hybrid image $x_{\mathrm{hybrid}}$ by combining their spectral components as follows:

where $\mathcal{F}$ denotes the 2D Discrete Fourier Transform, $\odot$ represents element-wise multiplication, and $M_r$ is a circular low-pass binary mask with radius $r = 50$. This operation preserves the semantic skeleton of the Common image while grafting onto it the exact high-frequency textures of the Jammed image. We define transferred toxicity as the fraction of random pairs $(x_j, x_c)$ for which the hybrid inherits the source jammer’s predicted class, i.e., $f_{\mathrm{std}}(x_{\mathrm{hybrid}}) = f_{\mathrm{std}}(x_j)$.

Results. As shown in Table 2, the results demonstrate a near-complete collapse of predictive influence upon transplantation. While suppressing high-frequency components yields a 31.8 percentage-point recovery on the native jammed inputs (§ 3.1), the transferred toxicity to a foreign host is only 0.17% (averaged over five random pairings), whereas reassembling a jammer from its own spectral components preserves its jammed prediction in 99.9% of cases.

Transfer scenario	Native toxicity (0% → X%)	Transferred toxicity (on new host)
$\mathcal{J} \to\mathcal{C}$	31.8% (0% → 31.8%)	0.17%

This result confirms that Natural Jammers are not independently “toxic” patterns. Their capacity to mislead the model is not an intrinsic property of the signal but emerges from a fragile resonance with the host image’s spatial structure. Once the spectral-spatial lock is broken by moving the HF signal to other host images, the feature becomes inefficient. This causal evidence shifts the understanding of non-robustness from a feature-level defect to a systemic interaction between spectrum and space.

Formal Definition: Spectral–Spatial Locking

The empirical evidence in the preceding sections reveals that natural non-robustness is neither a purely spectral defect nor a static spatial anomaly. Instead, it manifests as a structural dependency where the predictive utility of non-robust features is conditioned on the alignment of frequency and space. We formalize this mechanism as follows:

Definition 3.1 establishes that individual jammers are fragile configurations that collapse when their spectral–spatial lock is disrupted. However, this fragility presents a paradox: if specific locks are so easily broken, why does the aggregate size of the Jammed Set remain approximately stable across models and augmentations? This indicates a redistribution of non-robustness across the data manifold. We will explore this in the next section.

Substitution Paradox: Systemic Redistribution of Natural Non-Robustness

Section 3 established that many clean-image failures of standard models are consistent with spectral–spatial locking: interfering cues are fragile under mild spectral or geometric perturbations. This fragility raises a natural question: if individual jammers are so easily disrupted, why does non-robust vulnerability persist at scale?

This section provides system-level evidence that natural non-robustness is not confined to a small set of “bad images” but is a persistent outcome of the model–data interaction. We focus on three questions: Q1: Is the prevalence of $\mathcal{J}$ and $\mathcal{S}$ stable across architectures and robustness recipes? Q2: Are jammers/spurious identities intrinsic to images, or model-dependent? Q3: Under tiny semantics-preserving transformations, do these pathologies disappear or get redistributed?

Experimental Setup. All analyses are conducted on ImageNet validation ($N = 50{,}000$). For each architecture family, we use its corresponding standard model as $f_{\mathrm{std}}$ and a collection of robust models as candidate lenses $f_{\mathrm{rob}}$ that vary in threat model ($\ell_2$ or $\ell_\infty$) and budget $\epsilon$. Within each architecture block, $f_{\mathrm{std}}$ is fixed to that architecture’s standard model, and each robust model defines a quad-set partition $\mathcal{C}, \mathcal{S}, \mathcal{J}, \mathcal{G}$ according to § 2 on the original clean inputs.

Prevalence Across Architectures

We first establish a baseline fact that will serve as a foundation for the remainder of the section: natural jamming is not rare, nor specific to a single robustness recipe. Table 3 reports the quad-set partition induced by 17 robust lenses spanning CNNs and ViTs.

Configuration	Jammed ($\mathcal{J}$)	Spurious ($\mathcal{S}$)	Common ($\mathcal{C}$)	Uninformative ($\mathcal{G}$)	Rob. Acc.
ResNet-18	Standard Accuracy: 69.76%
$\ell_2$, $\epsilon= 0.5$	2,639 (5.28%)	4,786 (9.57%)	30,094 (60.19%)	12,481 (24.96%)	65.5%
$\ell_2$, $\epsilon= 1.0$	2,448 (4.90%)	6,168 (12.34%)	28,712 (57.42%)	12,672 (25.34%)	62.3%
$\ell_2$, $\epsilon= 3.0$	1,924 (3.85%)	10,240 (20.48%)	24,640 (49.28%)	13,196 (26.39%)	53.1%
$\ell_\infty$, $\epsilon= 1.0$	2,502 (5.00%)	5,659 (11.32%)	29,221 (58.44%)	12,618 (25.24%)	63.5%
$\ell_\infty$, $\epsilon= 4.0$	1,867 (3.73%)	10,497 (21.00%)	24,383 (48.77%)	13,253 (26.51%)	52.5%
ResNet-50	Standard Accuracy: 76.15%
$\ell_2$, $\epsilon= 0.5$	2,317 (4.63%)	3,803 (7.61%)	34,270 (68.54%)	9,610 (19.22%)	73.2%
$\ell_2$, $\epsilon= 1.0$	2,140 (4.28%)	5,002 (10.00%)	33,071 (66.14%)	9,787 (19.57%)	70.4%
$\ell_2$, $\epsilon= 3.0$	1,807 (3.61%)	8,463 (16.93%)	29,610 (59.22%)	10,120 (20.24%)	62.8%
$\ell_\infty$, $\epsilon= 1.0$	2,252 (4.50%)	4,306 (8.61%)	33,767 (67.53%)	9,675 (19.35%)	72.0%
$\ell_\infty$, $\epsilon= 4.0$	1,812 (3.62%)	7,948 (15.90%)	30,125 (60.25%)	10,115 (20.23%)	63.9%
WideResNet-50-2	Standard Accuracy: 78.48%
$\ell_2$, $\epsilon= 0.5$	2,202 (4.40%)	3,887 (7.77%)	35,354 (70.71%)	8,557 (17.11%)	75.1%
$\ell_2$, $\epsilon= 1.0$	2,167 (4.33%)	4,698 (9.40%)	34,543 (69.09%)	8,592 (17.18%)	73.4%
$\ell_2$, $\epsilon= 3.0$	1,910 (3.82%)	7,701 (15.40%)	31,540 (63.08%)	8,849 (17.70%)	66.9%
$\ell_\infty$, $\epsilon= 1.0$	2,232 (4.46%)	4,145 (8.29%)	35,096 (70.19%)	8,527 (17.05%)	74.7%
$\ell_\infty$, $\epsilon= 4.0$	2,000 (4.00%)	7,037 (14.07%)	32,204 (64.41%)	8,759 (17.52%)	68.4%
ViT-B/16	Standard Accuracy: 81.07%
$\ell_\infty$, $\epsilon= 1.0$	1,569 (3.14%)	5,467 (10.93%)	35,067 (70.13%)	7,897 (15.79%)	73.3%
ViT-B/32	Standard Accuracy: 75.91%
$\ell_\infty$, $\epsilon= 1.0$	1,828 (3.66%)	6,511 (13.02%)	31,443 (62.89%)	10,218 (20.44%)	66.5%

Across a wide range of architectures and robustness recipes, the Jammed Set consistently occupies a few percent of ImageNet validation set, and the Spurious Set occupies a larger but still substantial portion (roughly $\sim$7–20%, depending on $\epsilon$). Natural jamming is thus not an isolated artifact of a particular model or robustness configuration; across these lenses, $|\mathcal{J}|$ is relatively stable compared to $|\mathcal{S}|$.

Model-Specific Vulnerabilities

We next test whether jammed is an intrinsic property of the image or depends on the model used to define the partition. For two induced Jammed sets $\mathcal{J}_A$ and $\mathcal{J}_B$, we quantify agreement via Intersection-over-Union (IoU):

and analogously for $\mathcal{S}$.

Agreement is modest within architecture and low across architectures. Figure 3 shows that even within the same architecture, changing only the robustness budget yields moderate agreement (IoU $\approx$ 0.40–0.47). Across architectures, IoU collapses (typically $< 0.15$), and can be even smaller between CNNs and ViTs. These results indicate that which images are jammed/spurious depends strongly on model inductive bias and robustness recipe, rather than being image-intrinsic.

While IoU captures identity, confidence helps assess risk: jammed errors are often high-confidence for standard models. Table 4 reports mean softmax confidence of $f_{\mathrm{std}}$ and the corresponding robust lens $f_{\mathrm{rob}}$ on the induced sets. Across CNNs, $f_{\mathrm{std}}$ is typically more confident than $f_{\mathrm{rob}}$ on $\mathcal{J}$, consistent with a brittle but dominant cue overwhelming semantic evidence. ViTs show a reversal (robust confidence > standard confidence), which may reflect architecture-specific interfering cues; however, the ViT and CNN lenses also differ in training recipe, so this comparison cannot separate the two factors.

CONFIG	$\epsilon$	JAMMED ($\mathcal{J}$) RN-18	JAMMED ($\mathcal{J}$) RN-50	JAMMED ($\mathcal{J}$) WRN-50-2	SPURIOUS ($\mathcal{S}$) RN-18	SPURIOUS ($\mathcal{S}$) RN-50	SPURIOUS ($\mathcal{S}$) WRN-50-2
$\ell_2$	0.5	.457 / .440	.511 / .509	.545 / .556	.571 / .349	.624 / .412	.684 / .459
$\ell_2$	1.0	.470 / .402	.517 / .464	.555 / .531	.601 / .303	.661 / .368	.710 / .427
$\ell_2$	3.0	.491 / .318	.544 / .384	.574 / .449	.678 / .217	.726 / .267	.770 / .323
$\ell_\infty$	1.0	.468 / .418	.513 / .485	.554 / .550	.588 / .316	.643 / .382	.698 / .445
$\ell_\infty$	4.0	.490 / .319	.538 / .394	.577 / .464	.680 / .206	.717 / .270	.760 / .321
ViT-B/16 ($\ell_\infty$=1.0)		.504 / .701			.640 / .584
ViT-B/32 ($\ell_\infty$=1.0)		.447 / .670			.591 / .541

The Substitution Paradox

Section 3 demonstrated that individual jammed behaviors can be neutralized by mild spectral filtering or small geometric shifts, suggesting that jamming depends on fragile spectral–spatial alignment. We now formalize and test a stronger claim: tiny transformations can break many specific jammed identities, yet the overall prevalence of jamming remains approximately conserved.

Crop	$|\mathcal{J}_T|$	$|\mathcal{S}_T|$	$\mathcal{J}$	$\mathcal{S}$
0% (original)	2,140	5,002	1.00	1.00
2%	2,289	5,193	0.46	0.58
4%	2,268	5,208	0.42	0.54
6%	2,276	5,127	0.38	0.51
8%	2,291	5,159	0.35	0.49

and analogously $S_T$, $C_T$, $G_T$. We quantify (i) identity stability via $\operatorname{IoU}(\mathcal{J},\mathcal{J}_T)$, and (ii) prevalence stability via the transformed set sizes $|\mathcal{J}_T|$ and $|\mathcal{S}_T|$.

We apply minimal center crops removing only 2%, 4%, 6%, or 8% of edge pixels and resize back to $224 \times224$. For each crop ratio, we compute the transformed partition (Definition 4.2) and report both set sizes and IoU against the original partition.

Identity instability (structural fragility): Even a 2% crop yields $\operatorname{IoU}(\mathcal{J},\mathcal{J}_T) = 0.46$, implying that about one third of the original jammed images leave $\mathcal{J}$, and a similar number of new ones enter, under an almost imperceptible geometric perturbation. This supports the view that the underlying cues depend on fragile spatial alignment, consistent with spectral–spatial locking.

Prevalence stability (substitution): Despite the large change in identity, the transformed set sizes remain stable: $|\mathcal{J}_T| \approx2.2\mathrm{k}$ and $|\mathcal{S}_T| \approx5.1\mathrm{k}$ across crop ratios. This stability is approximate rather than exact: in a re-run that tracks individual images, a 2% crop moves 738 of the 2,140 jammers out of $\mathcal{J}$ and 873 new images in. Thus, breaking many specific locks does not reduce overall vulnerability; instead, new jammed/spurious identities emerge elsewhere. This resolves the tension raised by § 3: individual jammers are fragile, yet aggregate non-robustness persists as a systemic property and is redistributed under small transformations. Similar turnover appears under mild low-pass filtering and on six further datasets; our claim is limited to these tested mild transformations. In this view, natural non-robustness is a systemic “tax” that the model must pay to achieve high training accuracy.

We formalized and verified the Substitution Paradox: tiny semantics-preserving geometric changes drastically alter jammed/spurious identities while leaving their prevalence approximately unchanged.

Related Work

Adversarial Robustness and Non-robust Features

The vulnerability of deep neural networks to adversarial examples Szegedy et al. [ [33], Goodfellow et al. [ [9], Madry et al. [ [23] has been increasingly understood through the lens of data-driven features. Ilyas et al. [ [13] famously posited that adversarial examples arise from non-robust features—patterns that are highly predictive yet imperceptible to humans—and that these features are actively useful for standard classification. This framework has led to a prevailing view of non-robust features as “spurious shortcuts” Geirhos et al. [ [8], Shah et al. [ [30] that models exploit to achieve high standard accuracy, often resulting in an inherent trade-off between robustness and clean performance [35, 39, 25].

This view has since been actively debated. Kim et al. [ [16] show via information bottleneck that non-robust features are decomposable, with only a subset driving adversarial predictions. Li et al. [2023] argue that non-robust features are paradigm-specific shortcuts useful only under supervised objectives, not across learning paradigms. Crucially, however, both works study features under intentional adversarial perturbation—inputs explicitly crafted to expose model vulnerabilities.

We ask a fundamentally different question: can non-robust features mislead standard models on clean, unmodified natural images, without any perturbation? We identify a Jammed Set ($\mathcal{J}$) where robust models outperform standard models on standard test images, and provide causal evidence for Spectral–Spatial Locking as the driving mechanism. The antagonism we document is not injected by an attacker but arises spontaneously from the interaction between standard training and the spectral-spatial structure of natural data. Our Substitution Paradox further reveals this to be a redistributing property of the model–data geometry: a model pair defines a failure field $j(x)=\mathbf{1}[f_{\mathrm{std}}(x)\ne y\land f_{\mathrm{rob}}(x)=y]$ over natural images, which are commonly assumed to lie near a low-dimensional data manifold [6, 31], and small transformations move the support of $j$ while approximately preserving its mass. This provides a complementary framework to studies of natural adversarial or hard examples [11, 26, 22, 42] (], Li et al. 2024), which rely on curated atypical subsets rather than identifying a structural mechanism within standard uncurated data.

Spectral and Geometric Bias in Generalization

The inductive biases of CNNs are often characterized by their sensitivity to specific input frequencies and spatial structures. From a Fourier perspective, models frequently exhibit a spectral bias, relying on high-frequency components that are critical for generalization but lack robustness [38, 36, 5, 17, 37]. Simultaneously, the lack of geometric invariance—often caused by aliasing in downsampling layers Zhang [2019]—renders models fragile to even minor image transformations [2, 24]. This fragility is often linked to a "texture bias," where models prioritize local patterns over global shapes [7, 12].

Our work synthesizes these views through the mechanism of Spectral–spatial Locking, providing causal evidence that natural jamming is contingent on the precise alignment of frequency and space. Furthermore, we address the stability of these vulnerabilities, i.e., the Substitution Paradox where the aggregate rate of jamming remains stable across the data manifold. This observation relates to the broader problem of underspecification [3], where models with identical training objectives exhibit divergent and unpredictable failure modes under distribution shifts [34, 1, 28].

Conclusion

We show that non-robust features on clean natural images can be antagonistic: on a Jammed Set, robust models outperform standard models without modifying inputs. Using a standard-vs-robust partition of ImageNet-Val, we characterize Jammed and Spurious regimes and provide evidence for spectral–spatial locking via frequency filtering, geometric perturbations, and frequency grafting.

At the system level, jammed identities are highly model-dependent while prevalence remains stable, yielding the Substitution Paradox: small semantics-preserving transformations reshuffle jammed membership without reducing its overall rate.

Acknowledgments and Disclosure of Funding

This research / project is supported by ASTAR under its Japan-Singapore Joint Call: JST-ASTAR 2024 (Project ID: R2416IR139). This research is also supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award MOE-T2EP20125-0005). The authors declare no competing interests.