betaFeedback

WorksComputer Science

Representational Geometry as a Diagnostic for Adversarial Robustness

Abstract

Adversarial training is a common technique for making neural networks more robust to small, intentionally crafted input perturbations, and robustness is commonly measured through accuracy under attack. By exploring inter-network representations across different adversarial training objectives, we can analyze whether multiple networks organize their latent representations similarly. We study Multilayer Perceptrons trained on MNIST under three adversarial training regimes and compare the resulting population training dynamics and geometry to those of a vanilla network. We use M-PHATE to embed data samples across training epochs and examine how the geometry of the data changes during training under adversarial objectives. On penultimate-layer representations we measure displacement between clean and perturbed points, distributional distance through maximum mean discrepancy, and nearest-neighbor recovery. Our visualizations and quantitative analyses show that adversarial training leads to structural changes in how networks organize internal data representations, and robust accuracy and representational stability can rank the same defenses in opposite ways. These findings support a discussion of re-evaluating how practitioners approach the auditing of model robustness by treating latent representations and their training trajectories as diagnostic artifacts across sets of networks.

Introduction

Robustness is most commonly reported as a single scalar, without identifying how the structure of internal representations is organized, or how that organization formed during training. Adversarial attacks introduce small, carefully crafted perturbations to data samples with the aim of inducing misclassifications [1, 2]. Adversarial training is the standard method of defense for neural networks, where the network is trained on perturbed samples to learn to correctly classify attack samples [3]. However, multiple networks can achieve similar accuracy on perturbed samples, even if their internal representations differ structurally. As a scalar, accuracy lacks the power to report whether their representation geometries are truly different.

Differences in internal representational geometry stem largely from adversarial training, where clean accuracy is consequently sacrificed for robustness [4, 5] while also altering the input features that a network learns from [6]. As a result, the representations learned from robust networks are structurally different from those of a standard network [4, 7]. Given that robustness is largely a representational phenomenon, assessing defenses through a single behavioral metric strips researchers of the benefit of understanding the structural organization within a network that enables it to defend against an adversary. While existing work leverages similarity metrics [8, 9] to compare standard and robust representations, these methods are limited to static post-hoc analyses. As a result, they overlook the training dynamics in networks and how these shift under varying perturbation strengths across groups of networks.

In this work, we examine what adversarial training does inside a network, across regimes trained under varying adversarial objectives and attack strengths. Per-epoch traces and penultimate layer representations are computational artifacts produced through standard and adversarial training. Every network is examined with the same underlying test images, in their clean and perturbed forms, so the measured geometry reflects common data as arranged by each network, which allows networks themselves to be compared directly against each other. Across 45 networks under three adversarial training objectives and five perturbation budgets, we show that adversarial training reshapes representational geometry in ways that accuracy is unable to capture, with the largest reshaping belonging to the defense that scores best on accuracy. This disagreement widens as the perturbation budget grows and holds across each geometric measure computed.

The main contributions of this work are as follows: (1) we measure the representation geometry of a population of adversarially trained neural networks uniformly across attack objectives and varying perturbation budgets; (2) we show that geometric analyses rank network robustness differently than standard evaluations under threat-model-matched attacks; (3) we show that geometry and accuracy capture different aspects of robustness, since a defended network can keep accuracy high while its representations drift substantially when attacked.

Background

Let fθ:X→RCf_\theta: \mathcal{X} \to\mathbb{R}^{C} be a classifier with loss l(fθ(x),y)l(f_\theta(x), y). An adversarial attack creates a misclassification-inducing perturbation δ\delta constrained to ∥δ∥p≤ϵ\lVert\delta\rVert_p \le\epsilon where ϵ\epsilon is the perturbation budget and ∥⋅∥p\lVert\cdot\rVert_p is the ℓp\ell_p norm [1, 2]. In the white-box case, attacks are constructed using the input gradient g=∇xl(fθ(x),y)g = \nabla_x l(f_\theta(x), y), which provides the direction of the steepest increase in loss. We use single-step, white-box attacks that differ only in how gg is normalized before scaling by ϵ\epsilon. The fast gradient sign method (FGSM) [2] perturbs the input by ϵ⋅sign⁡(g)\epsilon\cdot\operatorname{sign}(g) moving every input value by ϵ\epsilon. The ℓ2\ell_2- and ℓ∞\ell_\infty-bounded attacks compute ϵ⋅g/∥g∥2\epsilon\cdot g/\lVert g\rVert_2 and ϵ⋅g/∥g∥∞\epsilon\cdot g/\lVert g\rVert_\infty, respectively. All scalings are clipped to [0,1][0,1]. A min-max objective is optimized during adversarial training, min⁡θE(x,y)[max⁡∥δ∥p≤ϵl(fθ(x+δ),y)]\min_\theta\mathbb{E}_{(x,y)}[\max_{\lVert\delta\rVert_p\le\epsilon} l(f_\theta(x+\delta),y)], where the inner maximization is approximated by an attack, while perturbed inputs are generated.

The maximum mean discrepancy (MMD) [10] measures distances between feature distributions in a Reproducing Kernel Hilbert Space (RKHS), which is used as an attack-agnostic measure of global representational difference. For the nn matched pairs, we compute the estimate MMD^2=1n2∑i,jk(xi,xj)+1m2∑i,jk(yi,yj)−2nm∑i,jk(xi,yj)\widehat{MMD}^{2} = \frac{1}{n^{2}}\sum_{i,j} k(x_i,x_j) + \frac{1}{m^{2}}\sum_{i,j} k(y_i,y_j) - \frac{2}{nm}\sum_{i,j} k(x_i,y_j), with Gaussian kernel bandwidth which is chosen per configuration using the median heuristic (further details in Appendix A.1).

M-PHATE [11], an extension of the PHATE dimensionality reduction method [12], tracks and visualizes representational dynamics across training epochs, embedding these high-dimensional network trajectories into a low-dimensional space while preserving local and global structures. Here we change the perspective and study how the geometry of data (i.e., samples), rather than hidden units, evolves throughout training. We describe the M-PHATE construction in § 3.2.

Related work: The standard defense for neural networks is adversarial training which is known to transform representational structures by sacrificing clean accuracy [4], increasing reliance on robust features [6], and reorganizing latent structures [13]. To measure the structural differences between trained networks with a similarity measure, CKA [8] provides a static, pairwise comparison post-hoc by comparing centered kernel matrices. SVCCA integrates singular value decomposition and canonical correlation analysis to analyze differences in representations [9]. To analyze the interand intra-unit dynamical characteristics of a network throughout training, M-PHATE [11] embeds the activation tensor as a single object in low-dimensional space. Setting dimensionality to two or three enables visualizing the dynamics. This places our work alongside an increasingly vital domain that treats trained networks themselves as data to be analyzed directly. These efforts include predicting model properties from weights [14, 15], compiling datasets of model populations [16], and developing architectures that operate on weight spaces [17]. While weights are not well-ordered, activations are, which makes matched-pair displacement between clean/perturbed input pairs tractable.

Setup and methods

Models and training

All experiments use MNIST [18] with a fixed sample of 1,000 training images and 2,000 test images. We compute geometric measurements on a fixed batch of 1,000 test images with indices consistent across all experiments, so only initialization and batch ordering vary across runs. The architecture is a three-layer fully connected MLP (784 →\to 128 →\to 64 →\to 10) with ReLU activations. Let h1∈R128h_1 \in\mathbb{R}^{128} and h2∈R64h_2 \in\mathbb{R}^{64} be the two hidden layers, where all geometric measurements are calculated on the penultimate layer h2h_2. We use fully connected MLPs so that representations are free from pooling or convolutional structure, and the training objective directly contributes to the measured geometry rather than an architectural prior. For that same reason, we use single-step attacks, though multi-step attack suites exist [19]. We train under four regimes: vanilla on clean data, and three adversarial regimes applying single-step attacks on every batch at each epoch. Each regime is trained across five perturbation budgets referred to as Extreme Low to Extreme High. Each network is trained on three seeds, resulting in a total population of 48 networks: 45 adversarially-trained and 3 vanilla. Training and attack details are expanded upon in Appendix § 6.6.

Trajectory and joint embeddings

PHATE [12] embeds high-dimensional data by representing the data as an affinity graph to capture local relationships, applying a diffusion operator to learn global structure, and reducing the dimensionality with MDS, resulting in a low-dimensional representation of the original data. M-PHATE [11] builds on this, treating networks as dynamical systems where each epoch becomes a single slice in a multislice graph between hidden units. To study how this local geometry evolves, M-PHATE connects a hidden unit to itself across all epochs through interslice edges in the multi-slice graph, allowing a diffusion process to capture temporal evolution in structural organization. While Gigante et al. [11] track hidden units, here we track test samples directly, making each embedded point a single image at a single training epoch. This corresponds to embedding the sample ×\times epoch modes of the activation tensor, rather than the hidden unit ×\times epoch modes (see Appendix).

Temporal analysis is especially critical in defense settings because standard and adversarial training may differ in the organization of the final representations they arrive at, and the path taken to form them. Final-epoch representations hold limited information power since they are unable to differentiate a network that reorganized early from one that drifted throughout, yet these two behaviors represent different learning processes. During training, the hidden activations of a test batch are recorded per-epoch to create a tensor of shape (50 epochs ×\times 1,000 samples ×\times 128 units) for h1h_1 and shape (50 ×\times 1,000 \times$ 64) for h2h_2, so a single embedding displays the entire training cycle as one continuous object. To compare clean and perturbed samples within a shared space, we stack penultimate layer representations of each test batch into a two-slice tensor of shape (2 ×\times 1,000 ×\times 64) using the multi-slice graph construction of M-PHATE [11], with each clean point linked to its perturbed sample. Since the interslice edge is a matched pair, each clean image remains linked to its perturbed counterpart with weight exp⁡{−(di/σ)2}\exp\{-(d_i/\sigma)^2\}, where did_i is the distance between the pair, and σ\sigma is a global bandwidth set to their mean distance. Following prior work using these embeddings directly as geometric representations [11, 20] all measurements in § 3.3 are computed in this space. We report results corresponding to using the raw h2h_2 activations in Appendix § 7.

Geometric analysis

We feed clean and perturbed test batches through each network, extracting their penultimate representations hicleanh_i^{\mathrm{clean}} and hiadvh_i^{\mathrm{adv}}, which are jointly embedded, as described in § 3.2. All three quantities below are computed on the resulting 3-dimensional M-PHATE coordinates. Our batches keep sample correspondence, so we measure matched pair quantities, described below.

Normalized displacement (dnormd_{\mathrm{norm}}) is the mean distance that the applied perturbation moves a point, expressed relative to the scale of that network’s own representation space. Since raw distances are not comparable across networks, we divide mean perturbation displacement by the within-network median to obtain a dimensionless quantity that measures how far a point moves after perturbation, relative to how far apart two arbitrary clean points lie. This yields a dimensionless, scale-invariant metric appropriate for cross-network comparison.

Distributional divergence is the measure of dissimilarity or difference between two distributions of data points. We use MMD with a Gaussian kernel to test whether the representations of clean and perturbed points are drawn from the same distribution. Since MMD^2\widehat{\mathrm{MMD}}^{2} provides a scalar output, distributional difference is formally characterized using a permutation test with Nperm=1000N_{\mathrm{perm}} = 1000 permutations. Kernel parameters are fit per configuration.

Neighborhood recovery requires no normalization, as we report the fraction of perturbed points whose clean counterpart appears within their k=10k = 10 nearest clean neighbors, making the measure invariant to any rescaling of the representation space.

Representational drift from adversarial training

We report our main findings below: testing networks on common attacks aligns with geometric measurement ranking about robustness and similarity among points; standard seen attack metrics rank training regimes opposite of geometric measurements; accuracy-based and geometric metrics diverge in specific cases, showing that the two metrics are not interchangeable as measures of robustness.

Measuring robustness with accuracy can coincide with geometric stability

Figure 1 reports clean accuracy of each trained network and its gap in accuracy against each attack along the same perturbation budget level: accuracyclean−accuracyrobust\mathrm{accuracy}_{\mathrm{clean}} - \mathrm{accuracy}_{\mathrm{robust}}. Clean accuracy remains stable across regimes, except for the FGSM-trained network, which exhibits a known instability that can come from single-step adversarial training with large budgets [21].

Comparing each network against the attack it was trained on gives a standard measure of robustness, shown within Figure 1 (Mean ± s.d. are shown in Appendix § 7). A clear pattern arises here: l∞l_{\infty} exhibits the smallest gap between clean accuracy and robust accuracy, across its own attack. This suggests, by standard measures, that it ranks as the most robust network in this population. Geometric measures suggest the opposite, ranking it last across the board. Across the three attack families, the l∞l_{\infty}-trained network exhibits the largest mean displacement at Medium, High, and Extreme High, at 0.095, 0.153, and 0.233 (Figure 2). It also shows the lowest mean kNN recovery proportions, 0.77, 0.55, and 0.30 (Figure 3 (a)), while the FGSM-trained network has the smallest displacement and highest recovery, with l2l_2 between them. Distributional discrepancy follows a similar pattern (Figure 3 (b)), revealing that the defended network with the most appealing robustness number is the one that has moved its representations furthest from where they started: a defense can rearrange how a network represents its data with no guarantee that accuracy will reflect it.

Clean accuracy and adversarial performance gaps across training perturbation budgets

Figure 1. Clean accuracy and adversarial performance gaps for FGSM, ℓ2\ell_{2} and ℓ∞\ell_{\infty} attacks across training perturbation budgets. A gap of 0 means clean accuracy matches accuracy under attack and larger gaps mean a larger drop under attack.

Normalized displacement between matched clean and perturbed representations in 3D M-PHATE space

Figure 2. Normalized displacement between matched clean and perturbed representations in 3D M-PHATE space; larger values mean the perturbation moved representations further relative to the network’s own scale. Rows: trained network. Columns: budget and attack used to perturb, ordered by ascending vanilla displacement. Each cell represents the network trained at the budget and perturbations of that attack at the same budget. Mean ± s.d., three seeds.

kNN recovery and MMD p-value heatmaps

Figure 3. (a) Proportion of perturbed test points whose clean counterpart lies within its ten nearest neighbors in 3D PHATE space. (b) MMD permutation test pp-values between clean and perturbed points in 3D PHATE space. Recovery near 1 and pp near 1 both mean the perturbation left the representation essentially unchanged. For both panels, rows denote the trained network and applied evaluation attack, and columns denote the perturbation budget. Mean ± s.d. across three seeds.

Figure 4 shows the 2-dimensional M-PHATE embeddings for hidden-layer activations over 50 epochs for the four regimes at the High perturbation budget. Compared to the vanilla network dynamics, all adversarially trained networks show a clear pattern in either a single layer or both layers: learning stability begins to collapse at earlier epochs. The first hidden layer remains more stable through all 50 epochs than the second, where representation movement becomes difficult to track; the FGSM regime appears the least sporadic of the defended networks and closely resembles the vanilla dynamics. All

M-PHATE embeddings of $h_1$ and $h_2$ across training epochs

Figure 4. M-PHATE embeddings of h1h_1 (top) and h2h_2 (bottom) across all 50 training epochs at the High budget, one network per column. Each panel embeds the full trajectory of activations for the clean test batch, colored by epoch.

M-PHATE embeddings are in Appendix § 8, where these behaviors vary across budgets, and give a qualitative view of clean versus perturbed representations.

Ranking robust networks varies with evaluation attacks

The vanilla network explains why the l∞l_\infty-trained network appears most robust under threat-model-matched evaluation: under the same nominal ϵ\epsilon, an undefended network loses 0.232 accuracy under FGSM perturbation, but only 0.055 under l∞l_\infty-bounded perturbation. Standard evaluations introduce bias into comparison since different attack types have different optimization objectives. Geometric measurements complement this by evaluating each network against each type of attack equally, providing a natural architecture- and attack-agnostic benchmark, enabling practitioners to compare populations of networks without strict constraints or requirements.

At Extreme High FGSM attack, the l∞l_\infty-trained network provides little benefit in accuracy, reducing the vanilla gap of 0.856 to 0.721, where the FGSM-trained network reaches a far lower gap of 0.326, yet it retains much of the geometric benefit, with displacement reduced from 0.414 to 0.300 against the FGSM-trained displacement of 0.206. Therefore, the measured geometric benefit of a defense is not proportional to the benefit that accuracy measures. We also note that no adversarially trained network has larger displacement than the vanilla network in any of the 45 cells, shown in Figure 2.

Discussion, limitations, and next steps

Across a group of 48 networks spanning four regimes (three adversarial and one standard) and five perturbation budgets, rankings imposed by conventional robustness metrics are not necessarily in agreement with their geometric structures. This nuanced relationship is attributed to the choice of evaluation attack used to quantify robustness. Scoring a defense against an attack it was trained on means that each network faces a test of different difficulty, whereas geometric measures enable cross-comparison across networks and test attacks without having to choose which attack represents a given network. For instance, our undefended network loses 0.232 accuracy under FGSM attack, but only 0.055 under the l∞l_\infty-bounded attack at the same nominal ϵ\epsilon, showing that evaluating defenses at equivalent ϵ\epsilon-values subjects models to tests of unequal difficulty. Geometry does not replace accuracy, since accuracy only calculates whether or not a decision boundary was crossed, while displacement, MMD, and kNN recovery report how representations shifted, whether their global distribution changed, and whether local neighborhoods survived. Reporting both distinguishes a defense that stabilized a network’s representations from one that only scored well against a favorable attack. We acknowledge the results presented reflect small MLPs on MNIST with single-step attacks, and are mainly derived from penultimate layer representations. Perturbation budgets were chosen to follow convention within the domain literature, matching on a nominal ϵ\epsilon, though extended study could compute budgets matched on applied perturbation magnitude. In future work, we will extend this analysis to convolutional- and transformer-based architectures [22, 23, 24] on CIFAR-10 and ImageNet, where representational structure is richer and robustness claims carry practical weight for researchers. Our results also indicate the geometry within neural network representations can be used for adversarial detection, using geometric shift as the basis for identifying perturbed inputs.

Extended methodological details

Maximum mean discrepancy

The maximum mean discrepancy (MMD) [10] can be used to measure the distance between two sets of representations through their mean embeddings in a Reproducing Kernel Hilbert Space (RKHS). Let X={xi}i=1nX = \{x_i\}_{i=1}^{n} and Y={yj}j=1mY = \{y_j\}_{j=1}^{m} be the clean and perturbed representations, respectively, with n=m=1000n=m=1000 and pooled as Z=[X,Y]Z=[X,Y]. We use a Gaussian kernel

k(u,v)=exp⁡(−γ∥u−v∥22) with γ=1[median⁡i,j∥zi−zj∥2]2+ϵ0k(u,v)=\exp\left(-\gamma\lVert u-v\rVert_2^2\right) \text{ with } \gamma= \frac{1}{\left[\operatorname{median}_{i,j}\lVert z_i-z_j\rVert_2\right]^2+\epsilon_0}

with the bandwidth chosen using the median heuristic and ϵ0=10−12\epsilon_0=10^{-12}. The kernel matrix is computed once and held fixed. Using the kernel matrix K∈R(n+m)×(n+m)K \in\mathbb{R}^{(n+m)\times(n+m)} of ZZ and the weight vector w=(1n,…,1n,−1m,…,−1m)Tw=\left(\frac{1}{n},\ldots,\frac{1}{n},-\frac{1}{m},\ldots,-\frac{1}{m}\right)^{\mathsf{T}}, the biased estimator of MMD is measured as:

MMD^2=wTKw=1n2∑i,i′k(xi,xi′)+1m2∑j,j′k(yj,yj′)−2nm∑i,jk(xi,yj).\widehat{\mathrm{MMD}}^2=w^{\mathsf{T}}Kw=\frac{1}{n^2}\sum_{i,i'}k(x_i,x_{i'})+\frac{1}{m^2}\sum_{j,j'}k(y_j,y_{j'})-\frac{2}{nm}\sum_{i,j}k(x_i,y_j).

Since this statistic is invariant to global rescaling of the representation space, comparison across networks is appropriate. To properly compare MMD scalars across networks, we assess statistical significance using a permutation test with N=1,000N=1,000 permutations. Each permutation π\pi shuffles the pooled indices and recomputes MMD^π2=wπTKwπ\widehat{\mathrm{MMD}}_{\pi}^{2}=w_{\pi}^{\mathsf{T}}Kw_{\pi}. The p-value is computed as

p=1N∑π1[MMD^π2≥MMD^2].p=\frac{1}{N}\sum_{\pi}1\left[\widehat{\mathrm{MMD}}_{\pi}^{2}\geq\widehat{\mathrm{MMD}}^{2}\right].

which is reported as 0.000 when no permutation satisfies the inequality, indicating p<0.001p<0.001 rather than exact zero.

Normalized displacement

Let hicleanh_i^{\mathrm{clean}} and hiadvh_i^{\mathrm{adv}} be the representations of the i-th clean example and its perturbed counterpart, in the space being measured. The normalized displacement is computed as

dnorm=1n∑i=1n∥hiclean−hiadv∥2median⁡i,j∥hiclean−hjclean∥2.d_{\mathrm{norm}}=\frac{\frac{1}{n}\sum_{i=1}^{n}\lVert h_i^{\mathrm{clean}}-h_i^{\mathrm{adv}}\rVert_2}{\operatorname{median}_{i,j}\lVert h_i^{\mathrm{clean}}-h_j^{\mathrm{clean}}\rVert_2}.

kNN recovery

For each perturbed representation, hiadvh_i^{\mathrm{adv}}, let Nk(hiadv)N_k(h_i^{\mathrm{adv}}) be the index set of its kk clean-representation nearest neighbors under Euclidean distance. The kNN recovery rate is

Rk=1n∑i=1n1[i∈Nk(hiadv)].R_k=\frac{1}{n}\sum_{i=1}^{n}1\left[i\in N_k(h_i^{\mathrm{adv}})\right].

where k=10 in this work. Rk=1R_k=1 means every single perturbed point’s clean counterpart is within its k nearest clean neighbors, and Rk=0R_k=0 means no perturbed point is within its k clean nearest neighbors.

PHATE and M-PHATE

PHATE is a powerful dimensionality reduction technique that optimizes diffusion maps and aims to preserve both global and local structure for data visualization [12]. It works by building a graph of data points, computing affinities to capture local relationships, and applying a diffusion operator to capture multiscale global structure. The resulting embedding maps potential distances into a low-dimensional space, maintaining the spatial organization of its high-dimensional counterpart. Built upon PHATE, M-PHATE extends this to time-variant high-dimensional data (i.e. network representations over epochs) [11]. A tensor of shape (epochs ×\times samples ×\times units) is formed into a multislice graph where every epoch is a "slice", where interslice edges are connected to the same unit across epochs and intraslice edges connected within epochs. Applying diffusion on this combined graph creates an embedding where the representational trajectory of a network throughout training can be seen as a continuous structure, analogous to treating networks as dynamical systems whose temporal artifacts evolve along a low-dimensional trajectory. For further details, see Moon et al. [12] and Gigante et al. [11].

Attacks

Let g=∇xl(fθ(x),y)g = \nabla_x l(f_\theta(x), y) be the input gradient of a single sample, and let Π[0,1]\Pi_{[0,1]} denote clamping to the valid pixel range. The three single step attacks used are

xFGSMadv=Π[0,1](x+ϵ⋅sign⁡(g))x_{\mathrm{FGSM}}^{\mathrm{adv}} = \Pi_{[0,1]} \left(x + \epsilon\cdot\operatorname{sign}(g)\right)
xl2adv=Π[0,1](x+ϵ⋅g∥g∥2+ϵ0)x_{l_2}^{\mathrm{adv}} = \Pi_{[0,1]} \left(x + \epsilon\cdot\frac{g}{\lVert g \rVert_2 + \epsilon_0}\right)
xl∞adv=Π[0,1](x+ϵ⋅g∥g∥∞+ϵ0)x_{l_\infty}^{\mathrm{adv}} = \Pi_{[0,1]} \left(x + \epsilon\cdot\frac{g}{\lVert g \rVert_\infty+ \epsilon_0}\right)

where norms are computed over the flattened image and ϵ0=10−12\epsilon_0 = 10^{-12}.

Network training and attack configurations

Table 1 displays the details of training configurations used in this paper. This is held constant throughout all work in the paper, with the only variation being the adversarial training regime and perturbation budgets. Table 2 displays perturbation budgets as fractions and decimals for each attack used. Table 3 shows the mean perturbation magnitudes for each attacked batch of 1000 images used for evaluation.

ItemValue
ArchitectureMLP, 784 →\to 128 →\to 64 →\to 10, ReLU
OptimizerAdam
Learning rate η\eta0.001
Mini-batch size BB64
Epochs50
Losscross-entropy
DatasetMNIST
Training subsample1000 images drawn without replacement from the 60,000 training images
Subsample drawnp.random.seed(42); np.random.choice(60000, 1000, replace=False)
Test subsamplefirst 2000 test images; evaluation uses the first 1000 of these
Subsample indicesfixed across all seeds and all budgets
Seeds0, 1, 2 (np.random.seed, torch.manual_seed)

Table 1. Training configuration, identical for all networks and budgets.

BudgetFGSM ϵ\epsilonl2l_2 ϵ\epsilonl∞l_\infty ϵ\epsilon
Extreme Low2/2550.0078430.1252/2550.007843
Low4/2550.0156860.2504/2550.015686
Medium8/2550.0313730.5008/2550.031373
High16/2550.0627451.00016/2550.062745
Extreme High32/2550.1254902.00032/2550.125490

Table 2. Perturbation budgets. FGSM and l∞l_\infty use the same pixel-scale ϵ\epsilon, given both as a fraction of 255 and as a decimal; l2l_2 uses a separate scale.

BudgetAttackε\varepsilonmean ∥δ∥2\lVert\delta\rVert_2mean ∥δ∥∞\lVert\delta\rVert_\inftymean ∥δ∥1\lVert\delta\rVert_1
Extreme LowFGSM0.0078430.16930.00783.6696
Extreme Lowl2l_20.1250000.09650.01691.4967
Extreme Lowl∞l_\infty0.0078430.04110.00710.6388
LowFGSM0.0156860.33480.01577.2238
Lowl2l_20.2500000.19180.03392.9683
Lowl∞l_\infty0.0156860.08200.01431.2742
MediumFGSM0.0313730.66550.031414.2656
Mediuml2l_20.5000000.38110.06775.8665
Mediuml∞l_\infty0.0313730.16310.02852.5302
HighFGSM0.0627451.32500.062728.2429
Highl2l_21.0000000.75790.134911.5892
Highl∞l_\infty0.0627450.32410.05705.0035
Extreme HighFGSM0.1254902.63360.125555.8307
Extreme Highl2l_22.0000001.50480.269022.8762
Extreme Highl∞l_\infty0.1254900.64450.11389.8859

Table 3. Measured perturbation magnitudes, averaged over the 1000-image evaluation batch. δ=xadv−x\delta= x_{\mathrm{adv}} - x is measured after clamping to [0,1][0,1], so these are the perturbations actually applied rather than the nominal budgets.

Complete results tables

Table 4 shows the complete cross-attack accuracy measurements of all networks across all training regimes. Tables 5 and 6 give the corresponding geometry in the raw h2h_2 space and in the 3-component M-PHATE embedding, each reporting the mean displacement, the clean–clean median used as the denominator, the resulting normalized displacement, MMD2MMD^2 and corresponding p-values.

BudgetTrainedCleanAccuracy under attack: FGSMAccuracy under attack: l2l_2Accuracy under attack: l∞l_\inftyGap (clean − attacked): FGSMGap (clean − attacked): l2l_2Gap (clean − attacked): l∞l_\infty
Extreme LowVanilla0.893 ± 0.0020.855 ± 0.0020.862 ± 0.0070.882 ± 0.0050.039 ± 0.0020.031 ± 0.0060.011 ± 0.003
Extreme LowFGSM0.900 ± 0.0030.869 ± 0.0070.871 ± 0.0060.890 ± 0.0040.031 ± 0.0040.030 ± 0.0030.010 ± 0.002
Extreme Lowl2l_20.892 ± 0.0050.863 ± 0.0030.867 ± 0.0030.882 ± 0.0020.030 ± 0.0050.025 ± 0.0040.010 ± 0.004
Extreme Lowl∞l_\infty0.895 ± 0.0070.862 ± 0.0010.868 ± 0.0040.884 ± 0.0070.033 ± 0.0080.027 ± 0.0040.011 ± 0.002
LowVanilla0.893 ± 0.0020.806 ± 0.0020.828 ± 0.0040.868 ± 0.0080.087 ± 0.0000.066 ± 0.0050.025 ± 0.008
LowFGSM0.907 ± 0.0040.851 ± 0.0050.854 ± 0.0050.890 ± 0.0040.056 ± 0.0020.053 ± 0.0030.018 ± 0.001
Lowl2l_20.898 ± 0.0100.843 ± 0.0080.852 ± 0.0070.881 ± 0.0110.055 ± 0.0090.046 ± 0.0060.017 ± 0.003
Lowl∞l_\infty0.895 ± 0.0070.830 ± 0.0050.841 ± 0.0040.874 ± 0.0010.065 ± 0.0030.054 ± 0.0030.021 ± 0.006
MediumVanilla0.893 ± 0.0020.661 ± 0.0190.725 ± 0.0150.838 ± 0.0040.232 ± 0.0200.168 ± 0.0160.055 ± 0.005
MediumFGSM0.909 ± 0.0040.804 ± 0.0110.802 ± 0.0080.878 ± 0.0070.105 ± 0.0100.107 ± 0.0070.030 ± 0.006
Mediuml2l_20.908 ± 0.0080.785 ± 0.0010.796 ± 0.0040.873 ± 0.0050.123 ± 0.0090.112 ± 0.0080.035 ± 0.003
Mediuml∞l_\infty0.899 ± 0.0040.746 ± 0.0120.771 ± 0.0080.855 ± 0.0050.153 ± 0.0130.129 ± 0.0100.044 ± 0.006
HighVanilla0.893 ± 0.0020.296 ± 0.0300.430 ± 0.0380.766 ± 0.0050.597 ± 0.0310.464 ± 0.0390.127 ± 0.006
HighFGSM0.907 ± 0.0030.721 ± 0.0140.668 ± 0.0220.866 ± 0.0060.186 ± 0.0120.239 ± 0.0210.041 ± 0.009
Highl2l_20.916 ± 0.0070.684 ± 0.0040.696 ± 0.0080.865 ± 0.0100.231 ± 0.0050.220 ± 0.0010.050 ± 0.004
Highl∞l_\infty0.909 ± 0.0020.531 ± 0.0170.589 ± 0.0200.821 ± 0.0060.378 ± 0.0170.320 ± 0.0200.089 ± 0.007
Extreme HighVanilla0.893 ± 0.0020.038 ± 0.0020.092 ± 0.0060.514 ± 0.0340.856 ± 0.0010.801 ± 0.0070.380 ± 0.035
Extreme HighFGSM0.859 ± 0.0080.533 ± 0.0140.389 ± 0.0330.756 ± 0.0190.326 ± 0.0060.470 ± 0.0250.102 ± 0.012
Extreme Highl2l_20.897 ± 0.0040.485 ± 0.0150.499 ± 0.0190.758 ± 0.0030.412 ± 0.0190.398 ± 0.0220.139 ± 0.004
Extreme Highl∞l_\infty0.914 ± 0.0040.192 ± 0.0230.248 ± 0.0190.741 ± 0.0040.721 ± 0.0210.666 ± 0.0150.173 ± 0.002

Table 4. Complete cross-attack accuracy grid. Every network is evaluated on the clean batch and under all three attacks at that budget; attacks are generated white-box against the network being evaluated. Mean ± s.d. over seeds 0, 1, 2.

Additional figures

M-PHATE embeddings across training regimes and adversarial budgets

Figure 5 shows the M-PHATE trajectories of all networks across 50 epochs. Clean test samples are fed through at every epoch. Throughout these additional examples, the pattern noted in the main text still holds: the first hidden layers (h1h_1) of each of these networks appears to have more stable and consistent trajectory evolution throughout training than the second hidden layers (h2h_2), whose trajectory is hard to follow from epoch to epoch. For the adversarially trained networks, this becomes more pronounced at larger perturbation budgets, where data points across epochs do not progress in a similar way as h1h_1. The vanilla row is identical across all budgets.

M-PHATE embeddings of training trajectories for layers $h_1$ and $h_2$ across perturbation budgets

Figure 5. M-PHATE embeddings of training trajectories for layers h1h_1 (top block) and h2h_2 (bottom block), with one row per network and one column per perturbation budget. Each point is one clean test image at one epoch, colored by epoch. The vanilla row is identical across all five columns, since that network never sees an attack.

Joint M-PHATE embeddings across training regimes and adversarial budgets

Figure 6–Figure 9 show the 2D M-PHATE embeddings of both clean and perturbed data points as penultimate layer representations (h2h_2) from each respective network, created after training has concluded. In Figure 6, within rows as perturbation budgets increase, so does the shift in perturbed point clusters from clean points and this is especially noticeable with classes for digits 0, 2, and 6. The ℓ∞\ell_\infty attack (Figure 9) also produces the least visible separation at every budget, compared to the FGSM and ℓ2\ell_2 attack (Figure 7 and Figure 8). The embeddings provided are best examined for qualitative patterns, as precise magnitudes and variations across seeds are given in Table 5 and Table 6.

Embeddings of the vanilla network’s penultimate layer

Figure 6. Embeddings of the vanilla network’s penultimate layer. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across attacks (rows) and perturbation budgets (columns).

Embeddings under the FGSM attack

Figure 7. Embeddings under the FGSM attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).

Table 5: Geometry in the raw 64-dimensional penultimate (h2) space. Mean displacement and median clean–clean pairwise distance are reported so normalized values can be recomputed as their ratio. Mean ± s.d. over seeds 0, 1, 2.

TrainedAttackMean Disp.Median Clean–CleanNorm. Disp.kNN Rec.MMD2^2pp
VanillaFGSM1.144 ± 0.04130.977 ± 0.7870.0370 ± 0.00211.000 ± 0.0000.00017 ± 0.000031.000 ± 0.000
Vanillal2l_20.890 ± 0.03830.977 ± 0.7870.0288 ± 0.00181.000 ± 0.0000.00007 ± 0.000021.000 ± 0.000
Vanillal∞l_\infty0.376 ± 0.01630.977 ± 0.7870.0122 ± 0.00081.000 ± 0.0000.00001 ± 0.000001.000 ± 0.000
FGSMFGSM0.871 ± 0.01230.196 ± 0.4490.0288 ± 0.00011.000 ± 0.0000.00011 ± 0.000011.000 ± 0.000
FGSMl2l_20.751 ± 0.00930.196 ± 0.4490.0249 ± 0.00011.000 ± 0.0000.00005 ± 0.000011.000 ± 0.000
FGSMl∞l_\infty0.289 ± 0.00630.196 ± 0.4490.0096 ± 0.00011.000 ± 0.0000.00001 ± 0.000001.000 ± 0.000
l2l_2FGSM0.949 ± 0.03430.568 ± 0.3220.0310 ± 0.00081.000 ± 0.0000.00013 ± 0.000021.000 ± 0.000
l2l_2l2l_20.782 ± 0.02430.568 ± 0.3220.0256 ± 0.00051.000 ± 0.0000.00006 ± 0.000011.000 ± 0.000
l2l_2l∞l_\infty0.328 ± 0.01130.568 ± 0.3220.0107 ± 0.00031.000 ± 0.0000.00001 ± 0.000001.000 ± 0.000
l∞l_\inftyFGSM0.961 ± 0.01130.135 ± 0.3400.0319 ± 0.00001.000 ± 0.0000.00012 ± 0.000001.000 ± 0.000
l∞l_\inftyl2l_20.783 ± 0.00830.135 ± 0.3400.0260 ± 0.00021.000 ± 0.0000.00006 ± 0.000001.000 ± 0.000
l∞l_\inftyl∞l_\infty0.326 ± 0.00430.135 ± 0.3400.0108 ± 0.00021.000 ± 0.0000.00001 ± 0.000001.000 ± 0.000
Extreme Low Budget
VanillaFGSM2.262 ± 0.08130.977 ± 0.7870.0731 ± 0.00411.000 ± 0.0000.00068 ± 0.000100.918 ± 0.065
Vanillal2l_21.768 ± 0.07430.977 ± 0.7870.0571 ± 0.00361.000 ± 0.0000.00029 ± 0.000061.000 ± 0.001
Vanillal∞l_\infty0.751 ± 0.03330.977 ± 0.7870.0243 ± 0.00151.000 ± 0.0000.00005 ± 0.000011.000 ± 0.000
FGSMFGSM1.545 ± 0.04330.158 ± 0.9240.0512 ± 0.00021.000 ± 0.0000.00037 ± 0.000040.999 ± 0.001
FGSMl2l_21.405 ± 0.03930.158 ± 0.9240.0466 ± 0.00021.000 ± 0.0000.00018 ± 0.000021.000 ± 0.000
FGSMl∞l_\infty0.528 ± 0.01630.158 ± 0.9240.0175 ± 0.00031.000 ± 0.0000.00003 ± 0.000001.000 ± 0.000
l2l_2FGSM1.716 ± 0.03330.483 ± 0.4240.0563 ± 0.00131.000 ± 0.0000.00044 ± 0.000070.996 ± 0.003
l2l_2l2l_21.454 ± 0.02230.483 ± 0.4240.0477 ± 0.00071.000 ± 0.0000.00019 ± 0.000031.000 ± 0.000
l2l_2l∞l_\infty0.602 ± 0.00930.483 ± 0.4240.0197 ± 0.00021.000 ± 0.0000.00003 ± 0.000001.000 ± 0.000
l∞l_\inftyFGSM1.792 ± 0.04830.046 ± 0.5040.0596 ± 0.00061.000 ± 0.0000.00046 ± 0.000030.994 ± 0.004
l∞l_\inftyl2l_21.482 ± 0.03630.046 ± 0.5040.0493 ± 0.00051.000 ± 0.0000.00019 ± 0.000021.000 ± 0.000
l∞l_\inftyl∞l_\infty0.619 ± 0.01130.046 ± 0.5040.0206 ± 0.00031.000 ± 0.0000.00003 ± 0.000001.000 ± 0.000
Medium Budget
VanillaFGSM4.499 ± 0.15830.977 ± 0.7870.1454 ± 0.00800.996 ± 0.0030.00263 ± 0.000350.017 ± 0.012
Vanillal2l_23.511 ± 0.14730.977 ± 0.7870.1135 ± 0.00710.999 ± 0.0010.00112 ± 0.000220.526 ± 0.198
Vanillal∞l_\infty1.493 ± 0.06530.977 ± 0.7870.0482 ± 0.00311.000 ± 0.0000.00021 ± 0.000041.000 ± 0.000
FGSMFGSM2.570 ± 0.05628.917 ± 0.5200.0889 ± 0.00161.000 ± 0.0000.00105 ± 0.000150.575 ± 0.140
FGSMl2l_22.519 ± 0.03728.917 ± 0.5200.0871 ± 0.00161.000 ± 0.0000.00063 ± 0.000100.948 ± 0.053
FGSMl∞l_\infty0.885 ± 0.01828.917 ± 0.5200.0306 ± 0.00071.000 ± 0.0000.00008 ± 0.000011.000 ± 0.000
l2l_2FGSM2.876 ± 0.06428.991 ± 0.3140.0992 ± 0.00141.000 ± 0.0000.00138 ± 0.000150.318 ± 0.111
l2l_2l2l_22.515 ± 0.05928.991 ± 0.3140.0867 ± 0.00131.000 ± 0.0000.00064 ± 0.000080.946 ± 0.040
l2l_2l∞l_\infty1.018 ± 0.02428.991 ± 0.3140.0351 ± 0.00051.000 ± 0.0000.00011 ± 0.000011.000 ± 0.000
l∞l_\inftyFGSM3.377 ± 0.09330.431 ± 0.5360.1110 ± 0.00111.000 ± 0.0000.00154 ± 0.000170.215 ± 0.029
l∞l_\inftyl2l_22.839 ± 0.06930.431 ± 0.5360.0933 ± 0.00081.000 ± 0.0000.00060 ± 0.000090.959 ± 0.031
l∞l_\inftyl∞l_\infty1.187 ± 0.02630.431 ± 0.5360.0390 ± 0.00031.000 ± 0.0000.00011 ± 0.000021.000 ± 0.000
High Budget
VanillaFGSM8.967 ± 0.29930.977 ± 0.7870.2897 ± 0.01520.874 ± 0.0630.00964 ± 0.001250.000 ± 0.000
Vanillal2l_26.963 ± 0.28530.977 ± 0.7870.2250 ± 0.01390.949 ± 0.0330.00410 ± 0.000820.000 ± 0.001
Vanillal∞l_\infty2.965 ± 0.12930.977 ± 0.7870.0958 ± 0.00611.000 ± 0.0010.00083 ± 0.000130.788 ± 0.140
FGSMFGSM3.557 ± 0.19423.117 ± 1.0950.1453 ± 0.00670.993 ± 0.0130.00320 ± 0.000810.009 ± 0.014
FGSMl2l_23.632 ± 0.13923.117 ± 1.0950.1572 ± 0.00570.986 ± 0.0160.00259 ± 0.000530.025 ± 0.026
FGSMl∞l_\infty1.130 ± 0.07023.117 ± 1.0950.0489 ± 0.00271.000 ± 0.0000.00028 ± 0.000071.000 ± 0.000
l2l_2FGSM3.959 ± 0.10923.396 ± 0.1410.1692 ± 0.00490.994 ± 0.0070.00438 ± 0.000770.001 ± 0.001
l2l_2l2l_23.618 ± 0.07023.396 ± 0.1410.1546 ± 0.00330.989 ± 0.0130.00242 ± 0.000450.033 ± 0.041
l2l_2l∞l_\infty1.438 ± 0.03623.396 ± 0.1410.0615 ± 0.00161.000 ± 0.0000.00043 ± 0.000090.998 ± 0.003
l∞l_\inftyFGSM6.088 ± 0.20929.736 ± 0.8140.2047 ± 0.00300.992 ± 0.0030.00574 ± 0.000640.000 ± 0.000
l∞l_\inftyl2l_25.210 ± 0.17129.736 ± 0.8140.1752 ± 0.00260.994 ± 0.0010.00223 ± 0.000240.040 ± 0.037
l∞l_\inftyl∞l_\infty2.137 ± 0.06529.736 ± 0.8140.0719 ± 0.00151.000 ± 0.0000.00041 ± 0.000040.997 ± 0.003
Extreme High Budget
VanillaFGSM17.525 ± 0.48730.977 ± 0.7870.5661 ± 0.02470.240 ± 0.0300.03412 ± 0.004570.000 ± 0.000
Vanillal2l_213.589 ± 0.50130.977 ± 0.7870.4390 ± 0.02500.508 ± 0.0520.01521 ± 0.003490.000 ± 0.000
Vanillal∞l_\infty5.885 ± 0.25330.977 ± 0.7870.1902 ± 0.01190.983 ± 0.0100.00308 ± 0.000490.006 ± 0.006
FGSMFGSM3.152 ± 0.18213.333 ± 0.6880.2364 ± 0.00680.945 ± 0.0320.00913 ± 0.001320.000 ± 0.000
FGSMl2l_23.887 ± 0.16913.333 ± 0.6880.2916 ± 0.00270.804 ± 0.0220.00891 ± 0.000940.000 ± 0.000
FGSMl∞l_\infty1.108 ± 0.07113.333 ± 0.6880.0831 ± 0.00261.000 ± 0.0000.00091 ± 0.000110.711 ± 0.111
l2l_2FGSM4.077 ± 0.06914.473 ± 0.3500.2817 ± 0.00230.887 ± 0.0140.01461 ± 0.000720.000 ± 0.000
l2l_2l2l_23.719 ± 0.04614.473 ± 0.3500.2570 ± 0.00350.908 ± 0.0030.00717 ± 0.000390.000 ± 0.000
l2l_2l∞l_\infty1.576 ± 0.01114.473 ± 0.3500.1089 ± 0.00190.995 ± 0.0060.00152 ± 0.000100.214 ± 0.055
l∞l_\inftyFGSM9.052 ± 0.40325.573 ± 1.7300.3543 ± 0.00840.762 ± 0.0360.01708 ± 0.000810.000 ± 0.000
l∞l_\inftyl2l_27.970 ± 0.32425.573 ± 1.7300.3120 ± 0.00820.858 ± 0.0380.00647 ± 0.000700.000 ± 0.000
l∞l_\inftyl∞l_\infty3.203 ± 0.12025.573 ± 1.7300.1254 ± 0.00400.997 ± 0.0040.00143 ± 0.000170.286 ± 0.117

Table 5.

TrainedAttackMean Disp.Median Clean–CleanNorm. Disp.kNN Rec.MMD2^{2}pp
Extreme Low Budget
VanillaFGSM0.00169 ± 0.000280.02985 ± 0.000330.0565 ± 0.00880.899 ± 0.0370.00125 ± 0.000760.432 ± 0.405
Vanillal2l_20.00150 ± 0.000330.02980 ± 0.000310.0503 ± 0.01050.912 ± 0.0390.00126 ± 0.000990.480 ± 0.448
Vanillal∞l_\infty0.00097 ± 0.000270.02969 ± 0.000340.0328 ± 0.00870.921 ± 0.0260.00087 ± 0.000590.588 ± 0.365
FGSMFGSM0.00113 ± 0.000150.02971 ± 0.000260.0381 ± 0.00500.953 ± 0.0180.00040 ± 0.000230.891 ± 0.139
FGSMl2l_20.00119 ± 0.000200.02969 ± 0.000250.0400 ± 0.00670.936 ± 0.0140.00059 ± 0.000420.755 ± 0.305
FGSMl∞l_\infty0.00076 ± 0.000230.02950 ± 0.000170.0259 ± 0.00770.936 ± 0.0290.00057 ± 0.000420.736 ± 0.232
l2l_2FGSM0.00129 ± 0.000230.02981 ± 0.000840.0434 ± 0.00680.945 ± 0.0280.00066 ± 0.000410.723 ± 0.270
l2l_2l2l_20.00116 ± 0.000210.02977 ± 0.000800.0389 ± 0.00620.946 ± 0.0270.00053 ± 0.000420.804 ± 0.278
l2l_2l∞l_\infty0.00080 ± 0.000270.02965 ± 0.000790.0271 ± 0.00900.939 ± 0.0270.00063 ± 0.000470.712 ± 0.249
l∞l_\inftyFGSM0.00128 ± 0.000110.03018 ± 0.000110.0425 ± 0.00360.939 ± 0.0200.00047 ± 0.000140.852 ± 0.105
l∞l_\inftyl2l_20.00117 ± 0.000170.03015 ± 0.000140.0388 ± 0.00540.941 ± 0.0310.00049 ± 0.000270.809 ± 0.188
l∞l_\inftyl∞l_\infty0.00056 ± 0.000090.03000 ± 0.000190.0187 ± 0.00290.970 ± 0.0240.00013 ± 0.000070.997 ± 0.004
Low Budget
VanillaFGSM0.00291 ± 0.000330.03005 ± 0.000460.0966 ± 0.00960.783 ± 0.0150.00270 ± 0.001350.123 ± 0.178
Vanillal2l_20.00256 ± 0.000330.02997 ± 0.000370.0853 ± 0.01000.817 ± 0.0290.00262 ± 0.001400.148 ± 0.213
Vanillal∞l_\infty0.00141 ± 0.000300.02980 ± 0.000320.0472 ± 0.00960.913 ± 0.0310.00126 ± 0.000840.438 ± 0.461
FGSMFGSM0.00192 ± 0.000190.03000 ± 0.000470.0640 ± 0.00560.913 ± 0.0300.00096 ± 0.000410.531 ± 0.278
FGSMl2l_20.00183 ± 0.000210.02997 ± 0.000480.0609 ± 0.00600.916 ± 0.0360.00088 ± 0.000420.573 ± 0.300
FGSMl∞l_\infty0.00081 ± 0.000140.02971 ± 0.000460.0271 ± 0.00440.971 ± 0.0350.00025 ± 0.000170.963 ± 0.055
l2l_2FGSM0.00217 ± 0.000270.03002 ± 0.000830.0723 ± 0.00730.875 ± 0.0320.00145 ± 0.000480.283 ± 0.154
l2l_2l2l_20.00192 ± 0.000180.02995 ± 0.000810.0640 ± 0.00450.902 ± 0.0300.00105 ± 0.000160.477 ± 0.099
l2l_2l∞l_\infty0.00098 ± 0.000090.02970 ± 0.000760.0328 ± 0.00220.952 ± 0.0170.00037 ± 0.000060.929 ± 0.032
l∞l_\inftyFGSM0.00210 ± 0.000030.03049 ± 0.000040.0716 ± 0.00990.875 ± 0.0080.00113 ± 0.000100.408 ± 0.062
l∞l_\inftyl2l_20.00194 ± 0.000130.03041 ± 0.000060.0637 ± 0.00400.906 ± 0.0160.00088 ± 0.000240.542 ± 0.165
l∞l_\inftyl∞l_\infty0.00088 ± 0.000120.03010 ± 0.000040.0292 ± 0.00400.976 ± 0.0340.00023 ± 0.000110.974 ± 0.035
Medium Budget
VanillaFGSM0.00492 ± 0.000390.03046 ± 0.000650.1613 ± 0.00960.523 ± 0.0050.00606 ± 0.002420.005 ± 0.008
Vanillal2l_20.00423 ± 0.000430.03023 ± 0.000520.1397 ± 0.01170.615 ± 0.0130.00507 ± 0.001310.010 ± 0.017
Vanillal∞l_\infty0.00236 ± 0.000290.02998 ± 0.000370.0786 ± 0.00870.855 ± 0.0280.00259 ± 0.001240.132 ± 0.194
FGSMFGSM0.00307 ± 0.001120.03028 ± 0.000320.1015 ± 0.00500.795 ± 0.0270.00202 ± 0.001130.112 ± 0.018
FGSMl2l_20.00303 ± 0.000160.03022 ± 0.000270.1001 ± 0.00590.779 ± 0.0310.00180 ± 0.000280.165 ± 0.063
FGSMl∞l_\infty0.00125 ± 0.001110.02988 ± 0.000270.0418 ± 0.00400.959 ± 0.0400.0045 ± 0.00120.876 ± 0.083
l2l_2FGSM0.00364 ± 0.000380.03062 ± 0.000290.1189 ± 0.01290.705 ± 0.0260.00368 ± 0.001410.020 ± 0.020
l2l_2l2l_20.00320 ± 0.000360.03039 ± 0.000320.1053 ± 0.01240.751 ± 0.0340.00240 ± 0.001020.121 ± 0.111
l2l_2l∞l_\infty0.00159 ± 0.000350.03001 ± 0.000210.0529 ± 0.01200.942 ± 0.0360.00096 ± 0.000780.592 ± 0.413
l∞l_\inftyFGSM0.00380 ± 0.000080.03086 ± 0.000200.1231 ± 0.00230.671 ± 0.0030.00315 ± 0.000880.018 ± 0.002
l∞l_\inftyl2l_20.00338 ± 0.000180.03070 ± 0.000230.1101 ± 0.00530.720 ± 0.0200.00233 ± 0.000460.072 ± 0.053
l∞l_\inftyl∞l_\infty0.00161 ± 0.000140.03025 ± 0.000240.0532 ± 0.00450.927 ± 0.0410.00069 ± 0.000200.703 ± 0.156
High Budget
VanillaFGSM0.00816 ± 0.000260.03087 ± 0.000480.2644 ± 0.00470.247 ± 0.0250.01396 ± 0.002500.000 ± 0.000
Vanillal2l_20.00692 ± 0.000370.03035 ± 0.000610.2278 ± 0.00810.348 ± 0.0200.01156 ± 0.001790.000 ± 0.000
Vanillal∞l_\infty0.00382 ± 0.000330.03022 ± 0.000550.1262 ± 0.00860.665 ± 0.0190.00461 ± 0.001820.014 ± 0.023
FGSMFGSM0.00436 ± 0.000050.03040 ± 0.000310.1436 ± 0.00300.562 ± 0.0190.00400 ± 0.00230.008 ± 0.003
FGSMl2l_20.00441 ± 0.000170.03040 ± 0.000270.1451 ± 0.00690.540 ± 0.0380.00351 ± 0.000870.015 ± 0.011
FGSMl∞l_\infty0.00173 ± 0.000110.02976 ± 0.000200.0582 ± 0.00410.948 ± 0.0150.00082 ± 0.000230.601 ± 0.164
l2l_2FGSM0.00505 ± 0.000420.03126 ± 0.000480.1615 ± 0.01190.477 ± 0.0630.00503 ± 0.001430.003 ± 0.004
l2l_2l2l_20.00458 ± 0.000370.03102 ± 0.000520.1477 ± 0.01030.504 ± 0.0660.00381 ± 0.001580.035 ± 0.057
l2l_2l∞l_\infty0.00221 ± 0.000170.03036 ± 0.000510.0727 ± 0.00420.883 ± 0.0340.00138 ± 0.000390.312 ± 0.149
l∞l_\inftyFGSM0.00601 ± 0.000160.03102 ± 0.000420.1937 ± 0.00340.383 ± 0.0420.00619 ± 0.001070.000 ± 0.001
l∞l_\inftyl2l_20.00536 ± 0.000110.03069 ± 0.000370.1746 ± 0.01410.436 ± 0.0390.00420 ± 0.000410.003 ± 0.002
l∞l_\inftyl∞l_\infty0.00272 ± 0.000080.03042 ± 0.000300.0894 ± 0.00260.837 ± 0.0190.00160 ± 0.000240.218 ± 0.080
Extreme High Budget
VanillaFGSM0.01251 ± 0.000330.03022 ± 0.000440.4140 ± 0.01440.108 ± 0.0120.02170 ± 0.000720.000 ± 0.000
Vanillal2l_20.01082 ± 0.000140.03021 ± 0.000500.3583 ± 0.00800.160 ± 0.0250.02063 ± 0.000630.000 ± 0.000
Vanillal∞l_\infty0.00616 ± 0.000320.03042 ± 0.000540.2024 ± 0.00700.399 ± 0.0110.00901 ± 0.002330.000 ± 0.000
FGSMFGSM0.00613 ± 0.000090.02984 ± 0.000610.2056 ± 0.00340.371 ± 0.0160.00576 ± 0.000550.001 ± 0.001
FGSMl2l_20.00745 ± 0.000250.03042 ± 0.000570.2450 ± 0.00380.291 ± 0.0310.00758 ± 0.001770.000 ± 0.000
FGSMl∞l_\infty0.00248 ± 0.000060.02978 ± 0.000460.0832 ± 0.00230.860 ± 0.0180.00136 ± 0.000140.284 ± 0.061
l2l_2FGSM0.00753 ± 0.000190.03220 ± 0.000390.2338 ± 0.00750.225 ± 0.0130.01125 ± 0.002540.000 ± 0.000
l2l_2l2l_20.00672 ± 0.000210.03197 ± 0.000380.2132 ± 0.00850.252 ± 0.0180.00758 ± 0.001980.000 ± 0.001
l2l_2l∞l_\infty0.00325 ± 0.000300.03110 ± 0.000240.1045 ± 0.00910.658 ± 0.0210.00236 ± 0.000450.078 ± 0.055
l∞l_\inftyFGSM0.00952 ± 0.000120.03171 ± 0.000420.3004 ± 0.00420.138 ± 0.0250.02139 ± 0.000510.000 ± 0.000
l∞l_\inftyl2l_20.00833 ± 0.000020.03145 ± 0.000170.2648 ± 0.00190.164 ± 0.0210.01402 ± 0.001930.000 ± 0.000
l∞l_\inftyl∞l_\infty0.00410 ± 0.000380.03075 ± 0.000250.1333 ± 0.01120.597 ± 0.0390.00363 ± 0.001600.030 ± 0.039

Table 6. Geometry in the 3-component M-PHATE embedding of the joint clean/perturbed tensor. Columns match Table 5. Mean ± s.d. over seeds 0, 1, 2.

Joint clean and perturbed embeddings under the L2 attack, with columns Vanilla, FGSM, L2, and Linf and rows Extreme Low, Low, Medium, High, and Extreme High

Figure 8. Embeddings under the l2l_2 attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).

Joint clean and perturbed embeddings under the $L_{\infty}$ attack

Figure 9. Embeddings under the l∞l_{\infty} attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).

Computing resources

All experiments were run on a single machine: one NVIDIA Tesla V100 (16 GB), a 16-core x86_64 CPU, and 64 GB of system memory.

Each of the five budget sets is one self-contained run covering 3 seeds ×\times 4 networks: 12 networks trained for 50 epochs. A run took roughly 20 minutes, and the five sets together took 101.9 minutes of wall-clock time. Network training is a minority of that cost: 26.2 minutes in total, against 35.0 minutes for the embeddings and 39.4 minutes for the permutation tests (0.9 minutes for the kNN and distance computations). Embedding, not training, dominates this pipeline, and the cost depends strongly on the number of slices. Averaged over exploratory runs, a 50-slice training-dynamics embedding takes 489 ±\pm 60 s, while a 2-slice clean-versus-perturbed embedding takes 124 ±\pm 30 s.

References

  1. [1]Christian Szegedy, Wojciech Zaremba, I. Sutskever, Joan Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. CoRR, December 2013.arxiv.org/abs/1312.6199
  2. [2]Ian J. Goodfellow, Jonathan Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. URL https://arxiv.org/abs/1412.6572.
  3. [3]Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks, September 2019.
  4. [4]Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness May Be at Odds with Accuracy, September 2019.arxiv.org/abs/1805.12152
  5. [5]Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019.arxiv.org/abs/1901.08573
  6. [6]Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In H. Wallace, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. 2019.arxiv.org/abs/1905.02175
  7. [7]Liwei Wang, Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hopcroft. Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation, November 2018.arxiv.org/abs/1810.11750
  8. [8]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. In Proceedings of the 36th International Conference on Machine Learning, pages 3519–3529, May 2019.arxiv.org/abs/1905.00414
  9. [9]Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.05806
  10. [10]Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25):723–773, 2012. ISSN 1533-7928.DOI
  11. [11]Scott Gigante, Adam S. Charles, Smita Krishnaswamy, and Gal Mishne. Visualizing the PHATE of Neural Networks. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1908.02831
  12. [12]Kevin R. Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B. Burkhardt, William S. Chen, Kristina Yim, Antonia van den Elzen, Matthew J. Hirn, Ronald R. Coifman, Natalia B. Ivanova, Guy Wolf, and Smita Krishnaswamy. Visualizing structure and transitions in high-dimensional biological data. Nature Biotechnology, 37(12):1482–1492, December 2019. ISSN 1546-1696. doi: 10.1038/s41587-019-0336-3.DOI
  13. [13]Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial Robustness as a Prior for Learned Representations, September 2019.arxiv.org/abs/1906.00945
  14. [14]Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting Neural Network Accuracy from Weights, April 2021.arxiv.org/abs/2002.11448
  15. [15]Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the Classifier: Dissecting the Weight Space of Neural Networks. In ECAI 2020, pages 1119–1126. 2020. doi: 10.3233/FAIA200209.DOI
  16. [16]Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i-Nieto, and Damian Borth. Model Zoos: A Dataset of Diverse Populations of Neural Network Models. Advances in Neural Information Processing Systems, 35:38134–38148, December 2022. doi: 10.52202/068431-2763.DOI
  17. [17]Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant Architectures for Learning in Deep Weight Spaces. In Proceedings of the 40th International Conference on Machine Learning, pages 25790–25816, July 2023.arxiv.org/abs/2301.12780
  18. [18]Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998. ISSN 1558-2256. doi: 10.1109/5.726791.
  19. [19]Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020.arxiv.org/abs/2003.01690
  20. [20]Jiancheng Xie, Lou C Voinov, Noga Mudrik, Gal Mishne, and Adam Shabti Charles. Multiway multislice PHATE: Visualizing hidden dynamics of RNNs through training. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=9Yr4V7iZsq.
  21. [21]Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BJxO4OEFvH.arxiv.org/abs/2001.03994
  22. [22]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015.arxiv.org/abs/1512.03385
  23. [23]Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Proceedings of the British Machine Vision Conference 2016, pages 87.1–87.12, York, UK, 2016. ISBN 978-1-901725-59-9. doi: 10.5244/C.30.87.DOI
  24. [24]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021.

Paper details

vlnc:W_jkgry6b4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents