WorksComputer Science
Representational Geometry as a Diagnostic for Adversarial Robustness
Abstract
Adversarial training is a common technique for making neural networks more robust to small, intentionally crafted input perturbations, and robustness is commonly measured through accuracy under attack. By exploring inter-network representations across different adversarial training objectives, we can analyze whether multiple networks organize their latent representations similarly. We study Multilayer Perceptrons trained on MNIST under three adversarial training regimes and compare the resulting population training dynamics and geometry to those of a vanilla network. We use M-PHATE to embed data samples across training epochs and examine how the geometry of the data changes during training under adversarial objectives. On penultimate-layer representations we measure displacement between clean and perturbed points, distributional distance through maximum mean discrepancy, and nearest-neighbor recovery. Our visualizations and quantitative analyses show that adversarial training leads to structural changes in how networks organize internal data representations, and robust accuracy and representational stability can rank the same defenses in opposite ways. These findings support a discussion of re-evaluating how practitioners approach the auditing of model robustness by treating latent representations and their training trajectories as diagnostic artifacts across sets of networks.
Introduction
Robustness is most commonly reported as a single scalar, without identifying how the structure of internal representations is organized, or how that organization formed during training. Adversarial attacks introduce small, carefully crafted perturbations to data samples with the aim of inducing misclassifications [1, 2]. Adversarial training is the standard method of defense for neural networks, where the network is trained on perturbed samples to learn to correctly classify attack samples [3]. However, multiple networks can achieve similar accuracy on perturbed samples, even if their internal representations differ structurally. As a scalar, accuracy lacks the power to report whether their representation geometries are truly different.
Differences in internal representational geometry stem largely from adversarial training, where clean accuracy is consequently sacrificed for robustness [4, 5] while also altering the input features that a network learns from [6]. As a result, the representations learned from robust networks are structurally different from those of a standard network [4, 7]. Given that robustness is largely a representational phenomenon, assessing defenses through a single behavioral metric strips researchers of the benefit of understanding the structural organization within a network that enables it to defend against an adversary. While existing work leverages similarity metrics [8, 9] to compare standard and robust representations, these methods are limited to static post-hoc analyses. As a result, they overlook the training dynamics in networks and how these shift under varying perturbation strengths across groups of networks.
In this work, we examine what adversarial training does inside a network, across regimes trained under varying adversarial objectives and attack strengths. Per-epoch traces and penultimate layer representations are computational artifacts produced through standard and adversarial training. Every network is examined with the same underlying test images, in their clean and perturbed forms, so the measured geometry reflects common data as arranged by each network, which allows networks themselves to be compared directly against each other. Across 45 networks under three adversarial training objectives and five perturbation budgets, we show that adversarial training reshapes representational geometry in ways that accuracy is unable to capture, with the largest reshaping belonging to the defense that scores best on accuracy. This disagreement widens as the perturbation budget grows and holds across each geometric measure computed.
The main contributions of this work are as follows: (1) we measure the representation geometry of a population of adversarially trained neural networks uniformly across attack objectives and varying perturbation budgets; (2) we show that geometric analyses rank network robustness differently than standard evaluations under threat-model-matched attacks; (3) we show that geometry and accuracy capture different aspects of robustness, since a defended network can keep accuracy high while its representations drift substantially when attacked.
Background
Let be a classifier with loss . An adversarial attack creates a misclassification-inducing perturbation constrained to where is the perturbation budget and is the norm [1, 2]. In the white-box case, attacks are constructed using the input gradient , which provides the direction of the steepest increase in loss. We use single-step, white-box attacks that differ only in how is normalized before scaling by . The fast gradient sign method (FGSM) [2] perturbs the input by moving every input value by . The - and -bounded attacks compute and , respectively. All scalings are clipped to . A min-max objective is optimized during adversarial training, , where the inner maximization is approximated by an attack, while perturbed inputs are generated.
The maximum mean discrepancy (MMD) [10] measures distances between feature distributions in a Reproducing Kernel Hilbert Space (RKHS), which is used as an attack-agnostic measure of global representational difference. For the matched pairs, we compute the estimate , with Gaussian kernel bandwidth which is chosen per configuration using the median heuristic (further details in Appendix A.1).
M-PHATE [11], an extension of the PHATE dimensionality reduction method [12], tracks and visualizes representational dynamics across training epochs, embedding these high-dimensional network trajectories into a low-dimensional space while preserving local and global structures. Here we change the perspective and study how the geometry of data (i.e., samples), rather than hidden units, evolves throughout training. We describe the M-PHATE construction in § 3.2.
Related work: The standard defense for neural networks is adversarial training which is known to transform representational structures by sacrificing clean accuracy [4], increasing reliance on robust features [6], and reorganizing latent structures [13]. To measure the structural differences between trained networks with a similarity measure, CKA [8] provides a static, pairwise comparison post-hoc by comparing centered kernel matrices. SVCCA integrates singular value decomposition and canonical correlation analysis to analyze differences in representations [9]. To analyze the interand intra-unit dynamical characteristics of a network throughout training, M-PHATE [11] embeds the activation tensor as a single object in low-dimensional space. Setting dimensionality to two or three enables visualizing the dynamics. This places our work alongside an increasingly vital domain that treats trained networks themselves as data to be analyzed directly. These efforts include predicting model properties from weights [14, 15], compiling datasets of model populations [16], and developing architectures that operate on weight spaces [17]. While weights are not well-ordered, activations are, which makes matched-pair displacement between clean/perturbed input pairs tractable.
Setup and methods
Models and training
All experiments use MNIST [18] with a fixed sample of 1,000 training images and 2,000 test images. We compute geometric measurements on a fixed batch of 1,000 test images with indices consistent across all experiments, so only initialization and batch ordering vary across runs. The architecture is a three-layer fully connected MLP (784 128 64 10) with ReLU activations. Let and be the two hidden layers, where all geometric measurements are calculated on the penultimate layer . We use fully connected MLPs so that representations are free from pooling or convolutional structure, and the training objective directly contributes to the measured geometry rather than an architectural prior. For that same reason, we use single-step attacks, though multi-step attack suites exist [19]. We train under four regimes: vanilla on clean data, and three adversarial regimes applying single-step attacks on every batch at each epoch. Each regime is trained across five perturbation budgets referred to as Extreme Low to Extreme High. Each network is trained on three seeds, resulting in a total population of 48 networks: 45 adversarially-trained and 3 vanilla. Training and attack details are expanded upon in Appendix § 6.6.
Trajectory and joint embeddings
PHATE [12] embeds high-dimensional data by representing the data as an affinity graph to capture local relationships, applying a diffusion operator to learn global structure, and reducing the dimensionality with MDS, resulting in a low-dimensional representation of the original data. M-PHATE [11] builds on this, treating networks as dynamical systems where each epoch becomes a single slice in a multislice graph between hidden units. To study how this local geometry evolves, M-PHATE connects a hidden unit to itself across all epochs through interslice edges in the multi-slice graph, allowing a diffusion process to capture temporal evolution in structural organization. While Gigante et al. [11] track hidden units, here we track test samples directly, making each embedded point a single image at a single training epoch. This corresponds to embedding the sample epoch modes of the activation tensor, rather than the hidden unit epoch modes (see Appendix).
Temporal analysis is especially critical in defense settings because standard and adversarial training may differ in the organization of the final representations they arrive at, and the path taken to form them. Final-epoch representations hold limited information power since they are unable to differentiate a network that reorganized early from one that drifted throughout, yet these two behaviors represent different learning processes. During training, the hidden activations of a test batch are recorded per-epoch to create a tensor of shape (50 epochs 1,000 samples 128 units) for and shape (50 1,000 \times$ 64) for , so a single embedding displays the entire training cycle as one continuous object. To compare clean and perturbed samples within a shared space, we stack penultimate layer representations of each test batch into a two-slice tensor of shape (2 1,000 64) using the multi-slice graph construction of M-PHATE [11], with each clean point linked to its perturbed sample. Since the interslice edge is a matched pair, each clean image remains linked to its perturbed counterpart with weight , where is the distance between the pair, and is a global bandwidth set to their mean distance. Following prior work using these embeddings directly as geometric representations [11, 20] all measurements in § 3.3 are computed in this space. We report results corresponding to using the raw activations in Appendix § 7.
Geometric analysis
We feed clean and perturbed test batches through each network, extracting their penultimate representations and , which are jointly embedded, as described in § 3.2. All three quantities below are computed on the resulting 3-dimensional M-PHATE coordinates. Our batches keep sample correspondence, so we measure matched pair quantities, described below.
Normalized displacement () is the mean distance that the applied perturbation moves a point, expressed relative to the scale of that network’s own representation space. Since raw distances are not comparable across networks, we divide mean perturbation displacement by the within-network median to obtain a dimensionless quantity that measures how far a point moves after perturbation, relative to how far apart two arbitrary clean points lie. This yields a dimensionless, scale-invariant metric appropriate for cross-network comparison.
Distributional divergence is the measure of dissimilarity or difference between two distributions of data points. We use MMD with a Gaussian kernel to test whether the representations of clean and perturbed points are drawn from the same distribution. Since provides a scalar output, distributional difference is formally characterized using a permutation test with permutations. Kernel parameters are fit per configuration.
Neighborhood recovery requires no normalization, as we report the fraction of perturbed points whose clean counterpart appears within their nearest clean neighbors, making the measure invariant to any rescaling of the representation space.
Representational drift from adversarial training
We report our main findings below: testing networks on common attacks aligns with geometric measurement ranking about robustness and similarity among points; standard seen attack metrics rank training regimes opposite of geometric measurements; accuracy-based and geometric metrics diverge in specific cases, showing that the two metrics are not interchangeable as measures of robustness.
Measuring robustness with accuracy can coincide with geometric stability
Figure 1 reports clean accuracy of each trained network and its gap in accuracy against each attack along the same perturbation budget level: . Clean accuracy remains stable across regimes, except for the FGSM-trained network, which exhibits a known instability that can come from single-step adversarial training with large budgets [21].
Comparing each network against the attack it was trained on gives a standard measure of robustness, shown within Figure 1 (Mean ± s.d. are shown in Appendix § 7). A clear pattern arises here: exhibits the smallest gap between clean accuracy and robust accuracy, across its own attack. This suggests, by standard measures, that it ranks as the most robust network in this population. Geometric measures suggest the opposite, ranking it last across the board. Across the three attack families, the -trained network exhibits the largest mean displacement at Medium, High, and Extreme High, at 0.095, 0.153, and 0.233 (Figure 2). It also shows the lowest mean kNN recovery proportions, 0.77, 0.55, and 0.30 (Figure 3 (a)), while the FGSM-trained network has the smallest displacement and highest recovery, with between them. Distributional discrepancy follows a similar pattern (Figure 3 (b)), revealing that the defended network with the most appealing robustness number is the one that has moved its representations furthest from where they started: a defense can rearrange how a network represents its data with no guarantee that accuracy will reflect it.

Figure 1. Clean accuracy and adversarial performance gaps for FGSM, and attacks across training perturbation budgets. A gap of 0 means clean accuracy matches accuracy under attack and larger gaps mean a larger drop under attack.

Figure 2. Normalized displacement between matched clean and perturbed representations in 3D M-PHATE space; larger values mean the perturbation moved representations further relative to the network’s own scale. Rows: trained network. Columns: budget and attack used to perturb, ordered by ascending vanilla displacement. Each cell represents the network trained at the budget and perturbations of that attack at the same budget. Mean ± s.d., three seeds.

Figure 3. (a) Proportion of perturbed test points whose clean counterpart lies within its ten nearest neighbors in 3D PHATE space. (b) MMD permutation test -values between clean and perturbed points in 3D PHATE space. Recovery near 1 and near 1 both mean the perturbation left the representation essentially unchanged. For both panels, rows denote the trained network and applied evaluation attack, and columns denote the perturbation budget. Mean ± s.d. across three seeds.
Figure 4 shows the 2-dimensional M-PHATE embeddings for hidden-layer activations over 50 epochs for the four regimes at the High perturbation budget. Compared to the vanilla network dynamics, all adversarially trained networks show a clear pattern in either a single layer or both layers: learning stability begins to collapse at earlier epochs. The first hidden layer remains more stable through all 50 epochs than the second, where representation movement becomes difficult to track; the FGSM regime appears the least sporadic of the defended networks and closely resembles the vanilla dynamics. All

Figure 4. M-PHATE embeddings of (top) and (bottom) across all 50 training epochs at the High budget, one network per column. Each panel embeds the full trajectory of activations for the clean test batch, colored by epoch.
M-PHATE embeddings are in Appendix § 8, where these behaviors vary across budgets, and give a qualitative view of clean versus perturbed representations.
Ranking robust networks varies with evaluation attacks
The vanilla network explains why the -trained network appears most robust under threat-model-matched evaluation: under the same nominal , an undefended network loses 0.232 accuracy under FGSM perturbation, but only 0.055 under -bounded perturbation. Standard evaluations introduce bias into comparison since different attack types have different optimization objectives. Geometric measurements complement this by evaluating each network against each type of attack equally, providing a natural architecture- and attack-agnostic benchmark, enabling practitioners to compare populations of networks without strict constraints or requirements.
At Extreme High FGSM attack, the -trained network provides little benefit in accuracy, reducing the vanilla gap of 0.856 to 0.721, where the FGSM-trained network reaches a far lower gap of 0.326, yet it retains much of the geometric benefit, with displacement reduced from 0.414 to 0.300 against the FGSM-trained displacement of 0.206. Therefore, the measured geometric benefit of a defense is not proportional to the benefit that accuracy measures. We also note that no adversarially trained network has larger displacement than the vanilla network in any of the 45 cells, shown in Figure 2.
Discussion, limitations, and next steps
Across a group of 48 networks spanning four regimes (three adversarial and one standard) and five perturbation budgets, rankings imposed by conventional robustness metrics are not necessarily in agreement with their geometric structures. This nuanced relationship is attributed to the choice of evaluation attack used to quantify robustness. Scoring a defense against an attack it was trained on means that each network faces a test of different difficulty, whereas geometric measures enable cross-comparison across networks and test attacks without having to choose which attack represents a given network. For instance, our undefended network loses 0.232 accuracy under FGSM attack, but only 0.055 under the -bounded attack at the same nominal , showing that evaluating defenses at equivalent -values subjects models to tests of unequal difficulty. Geometry does not replace accuracy, since accuracy only calculates whether or not a decision boundary was crossed, while displacement, MMD, and kNN recovery report how representations shifted, whether their global distribution changed, and whether local neighborhoods survived. Reporting both distinguishes a defense that stabilized a network’s representations from one that only scored well against a favorable attack. We acknowledge the results presented reflect small MLPs on MNIST with single-step attacks, and are mainly derived from penultimate layer representations. Perturbation budgets were chosen to follow convention within the domain literature, matching on a nominal , though extended study could compute budgets matched on applied perturbation magnitude. In future work, we will extend this analysis to convolutional- and transformer-based architectures [22, 23, 24] on CIFAR-10 and ImageNet, where representational structure is richer and robustness claims carry practical weight for researchers. Our results also indicate the geometry within neural network representations can be used for adversarial detection, using geometric shift as the basis for identifying perturbed inputs.
Extended methodological details
Maximum mean discrepancy
The maximum mean discrepancy (MMD) [10] can be used to measure the distance between two sets of representations through their mean embeddings in a Reproducing Kernel Hilbert Space (RKHS). Let and be the clean and perturbed representations, respectively, with and pooled as . We use a Gaussian kernel
with the bandwidth chosen using the median heuristic and . The kernel matrix is computed once and held fixed. Using the kernel matrix of and the weight vector , the biased estimator of MMD is measured as:
Since this statistic is invariant to global rescaling of the representation space, comparison across networks is appropriate. To properly compare MMD scalars across networks, we assess statistical significance using a permutation test with permutations. Each permutation shuffles the pooled indices and recomputes . The p-value is computed as
which is reported as 0.000 when no permutation satisfies the inequality, indicating rather than exact zero.
Normalized displacement
Let and be the representations of the i-th clean example and its perturbed counterpart, in the space being measured. The normalized displacement is computed as
kNN recovery
For each perturbed representation, , let be the index set of its clean-representation nearest neighbors under Euclidean distance. The kNN recovery rate is
where k=10 in this work. means every single perturbed point’s clean counterpart is within its k nearest clean neighbors, and means no perturbed point is within its k clean nearest neighbors.
PHATE and M-PHATE
PHATE is a powerful dimensionality reduction technique that optimizes diffusion maps and aims to preserve both global and local structure for data visualization [12]. It works by building a graph of data points, computing affinities to capture local relationships, and applying a diffusion operator to capture multiscale global structure. The resulting embedding maps potential distances into a low-dimensional space, maintaining the spatial organization of its high-dimensional counterpart. Built upon PHATE, M-PHATE extends this to time-variant high-dimensional data (i.e. network representations over epochs) [11]. A tensor of shape (epochs samples units) is formed into a multislice graph where every epoch is a "slice", where interslice edges are connected to the same unit across epochs and intraslice edges connected within epochs. Applying diffusion on this combined graph creates an embedding where the representational trajectory of a network throughout training can be seen as a continuous structure, analogous to treating networks as dynamical systems whose temporal artifacts evolve along a low-dimensional trajectory. For further details, see Moon et al. [12] and Gigante et al. [11].
Attacks
Let be the input gradient of a single sample, and let denote clamping to the valid pixel range. The three single step attacks used are
where norms are computed over the flattened image and .
Network training and attack configurations
Table 1 displays the details of training configurations used in this paper. This is held constant throughout all work in the paper, with the only variation being the adversarial training regime and perturbation budgets. Table 2 displays perturbation budgets as fractions and decimals for each attack used. Table 3 shows the mean perturbation magnitudes for each attacked batch of 1000 images used for evaluation.
| Item | Value |
| Architecture | MLP, 784 128 64 10, ReLU |
| Optimizer | Adam |
| Learning rate | 0.001 |
| Mini-batch size | 64 |
| Epochs | 50 |
| Loss | cross-entropy |
| Dataset | MNIST |
| Training subsample | 1000 images drawn without replacement from the 60,000 training images |
| Subsample draw | np.random.seed(42); np.random.choice(60000, 1000, replace=False) |
| Test subsample | first 2000 test images; evaluation uses the first 1000 of these |
| Subsample indices | fixed across all seeds and all budgets |
| Seeds | 0, 1, 2 (np.random.seed, torch.manual_seed) |
Table 1. Training configuration, identical for all networks and budgets.
| Budget | FGSM | ||||
| Extreme Low | 2/255 | 0.007843 | 0.125 | 2/255 | 0.007843 |
| Low | 4/255 | 0.015686 | 0.250 | 4/255 | 0.015686 |
| Medium | 8/255 | 0.031373 | 0.500 | 8/255 | 0.031373 |
| High | 16/255 | 0.062745 | 1.000 | 16/255 | 0.062745 |
| Extreme High | 32/255 | 0.125490 | 2.000 | 32/255 | 0.125490 |
Table 2. Perturbation budgets. FGSM and use the same pixel-scale , given both as a fraction of 255 and as a decimal; uses a separate scale.
| Budget | Attack | mean | mean | mean | |
| Extreme Low | FGSM | 0.007843 | 0.1693 | 0.0078 | 3.6696 |
| Extreme Low | 0.125000 | 0.0965 | 0.0169 | 1.4967 | |
| Extreme Low | 0.007843 | 0.0411 | 0.0071 | 0.6388 | |
| Low | FGSM | 0.015686 | 0.3348 | 0.0157 | 7.2238 |
| Low | 0.250000 | 0.1918 | 0.0339 | 2.9683 | |
| Low | 0.015686 | 0.0820 | 0.0143 | 1.2742 | |
| Medium | FGSM | 0.031373 | 0.6655 | 0.0314 | 14.2656 |
| Medium | 0.500000 | 0.3811 | 0.0677 | 5.8665 | |
| Medium | 0.031373 | 0.1631 | 0.0285 | 2.5302 | |
| High | FGSM | 0.062745 | 1.3250 | 0.0627 | 28.2429 |
| High | 1.000000 | 0.7579 | 0.1349 | 11.5892 | |
| High | 0.062745 | 0.3241 | 0.0570 | 5.0035 | |
| Extreme High | FGSM | 0.125490 | 2.6336 | 0.1255 | 55.8307 |
| Extreme High | 2.000000 | 1.5048 | 0.2690 | 22.8762 | |
| Extreme High | 0.125490 | 0.6445 | 0.1138 | 9.8859 |
Table 3. Measured perturbation magnitudes, averaged over the 1000-image evaluation batch. is measured after clamping to , so these are the perturbations actually applied rather than the nominal budgets.
Complete results tables
Table 4 shows the complete cross-attack accuracy measurements of all networks across all training regimes. Tables 5 and 6 give the corresponding geometry in the raw space and in the 3-component M-PHATE embedding, each reporting the mean displacement, the clean–clean median used as the denominator, the resulting normalized displacement, and corresponding p-values.
| Budget | Trained | Clean | Accuracy under attack: FGSM | Accuracy under attack: | Accuracy under attack: | Gap (clean − attacked): FGSM | Gap (clean − attacked): | Gap (clean − attacked): |
| Extreme Low | Vanilla | 0.893 ± 0.002 | 0.855 ± 0.002 | 0.862 ± 0.007 | 0.882 ± 0.005 | 0.039 ± 0.002 | 0.031 ± 0.006 | 0.011 ± 0.003 |
| Extreme Low | FGSM | 0.900 ± 0.003 | 0.869 ± 0.007 | 0.871 ± 0.006 | 0.890 ± 0.004 | 0.031 ± 0.004 | 0.030 ± 0.003 | 0.010 ± 0.002 |
| Extreme Low | 0.892 ± 0.005 | 0.863 ± 0.003 | 0.867 ± 0.003 | 0.882 ± 0.002 | 0.030 ± 0.005 | 0.025 ± 0.004 | 0.010 ± 0.004 | |
| Extreme Low | 0.895 ± 0.007 | 0.862 ± 0.001 | 0.868 ± 0.004 | 0.884 ± 0.007 | 0.033 ± 0.008 | 0.027 ± 0.004 | 0.011 ± 0.002 | |
| Low | Vanilla | 0.893 ± 0.002 | 0.806 ± 0.002 | 0.828 ± 0.004 | 0.868 ± 0.008 | 0.087 ± 0.000 | 0.066 ± 0.005 | 0.025 ± 0.008 |
| Low | FGSM | 0.907 ± 0.004 | 0.851 ± 0.005 | 0.854 ± 0.005 | 0.890 ± 0.004 | 0.056 ± 0.002 | 0.053 ± 0.003 | 0.018 ± 0.001 |
| Low | 0.898 ± 0.010 | 0.843 ± 0.008 | 0.852 ± 0.007 | 0.881 ± 0.011 | 0.055 ± 0.009 | 0.046 ± 0.006 | 0.017 ± 0.003 | |
| Low | 0.895 ± 0.007 | 0.830 ± 0.005 | 0.841 ± 0.004 | 0.874 ± 0.001 | 0.065 ± 0.003 | 0.054 ± 0.003 | 0.021 ± 0.006 | |
| Medium | Vanilla | 0.893 ± 0.002 | 0.661 ± 0.019 | 0.725 ± 0.015 | 0.838 ± 0.004 | 0.232 ± 0.020 | 0.168 ± 0.016 | 0.055 ± 0.005 |
| Medium | FGSM | 0.909 ± 0.004 | 0.804 ± 0.011 | 0.802 ± 0.008 | 0.878 ± 0.007 | 0.105 ± 0.010 | 0.107 ± 0.007 | 0.030 ± 0.006 |
| Medium | 0.908 ± 0.008 | 0.785 ± 0.001 | 0.796 ± 0.004 | 0.873 ± 0.005 | 0.123 ± 0.009 | 0.112 ± 0.008 | 0.035 ± 0.003 | |
| Medium | 0.899 ± 0.004 | 0.746 ± 0.012 | 0.771 ± 0.008 | 0.855 ± 0.005 | 0.153 ± 0.013 | 0.129 ± 0.010 | 0.044 ± 0.006 | |
| High | Vanilla | 0.893 ± 0.002 | 0.296 ± 0.030 | 0.430 ± 0.038 | 0.766 ± 0.005 | 0.597 ± 0.031 | 0.464 ± 0.039 | 0.127 ± 0.006 |
| High | FGSM | 0.907 ± 0.003 | 0.721 ± 0.014 | 0.668 ± 0.022 | 0.866 ± 0.006 | 0.186 ± 0.012 | 0.239 ± 0.021 | 0.041 ± 0.009 |
| High | 0.916 ± 0.007 | 0.684 ± 0.004 | 0.696 ± 0.008 | 0.865 ± 0.010 | 0.231 ± 0.005 | 0.220 ± 0.001 | 0.050 ± 0.004 | |
| High | 0.909 ± 0.002 | 0.531 ± 0.017 | 0.589 ± 0.020 | 0.821 ± 0.006 | 0.378 ± 0.017 | 0.320 ± 0.020 | 0.089 ± 0.007 | |
| Extreme High | Vanilla | 0.893 ± 0.002 | 0.038 ± 0.002 | 0.092 ± 0.006 | 0.514 ± 0.034 | 0.856 ± 0.001 | 0.801 ± 0.007 | 0.380 ± 0.035 |
| Extreme High | FGSM | 0.859 ± 0.008 | 0.533 ± 0.014 | 0.389 ± 0.033 | 0.756 ± 0.019 | 0.326 ± 0.006 | 0.470 ± 0.025 | 0.102 ± 0.012 |
| Extreme High | 0.897 ± 0.004 | 0.485 ± 0.015 | 0.499 ± 0.019 | 0.758 ± 0.003 | 0.412 ± 0.019 | 0.398 ± 0.022 | 0.139 ± 0.004 | |
| Extreme High | 0.914 ± 0.004 | 0.192 ± 0.023 | 0.248 ± 0.019 | 0.741 ± 0.004 | 0.721 ± 0.021 | 0.666 ± 0.015 | 0.173 ± 0.002 |
Table 4. Complete cross-attack accuracy grid. Every network is evaluated on the clean batch and under all three attacks at that budget; attacks are generated white-box against the network being evaluated. Mean ± s.d. over seeds 0, 1, 2.
Additional figures
M-PHATE embeddings across training regimes and adversarial budgets
Figure 5 shows the M-PHATE trajectories of all networks across 50 epochs. Clean test samples are fed through at every epoch. Throughout these additional examples, the pattern noted in the main text still holds: the first hidden layers () of each of these networks appears to have more stable and consistent trajectory evolution throughout training than the second hidden layers (), whose trajectory is hard to follow from epoch to epoch. For the adversarially trained networks, this becomes more pronounced at larger perturbation budgets, where data points across epochs do not progress in a similar way as . The vanilla row is identical across all budgets.

Figure 5. M-PHATE embeddings of training trajectories for layers (top block) and (bottom block), with one row per network and one column per perturbation budget. Each point is one clean test image at one epoch, colored by epoch. The vanilla row is identical across all five columns, since that network never sees an attack.
Joint M-PHATE embeddings across training regimes and adversarial budgets
Figure 6–Figure 9 show the 2D M-PHATE embeddings of both clean and perturbed data points as penultimate layer representations () from each respective network, created after training has concluded. In Figure 6, within rows as perturbation budgets increase, so does the shift in perturbed point clusters from clean points and this is especially noticeable with classes for digits 0, 2, and 6. The attack (Figure 9) also produces the least visible separation at every budget, compared to the FGSM and attack (Figure 7 and Figure 8). The embeddings provided are best examined for qualitative patterns, as precise magnitudes and variations across seeds are given in Table 5 and Table 6.

Figure 6. Embeddings of the vanilla network’s penultimate layer. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across attacks (rows) and perturbation budgets (columns).

Figure 7. Embeddings under the FGSM attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).
Table 5: Geometry in the raw 64-dimensional penultimate (h2) space. Mean displacement and median clean–clean pairwise distance are reported so normalized values can be recomputed as their ratio. Mean ± s.d. over seeds 0, 1, 2.
| Trained | Attack | Mean Disp. | Median Clean–Clean | Norm. Disp. | kNN Rec. | MMD | |
| Vanilla | FGSM | 1.144 ± 0.041 | 30.977 ± 0.787 | 0.0370 ± 0.0021 | 1.000 ± 0.000 | 0.00017 ± 0.00003 | 1.000 ± 0.000 |
| Vanilla | 0.890 ± 0.038 | 30.977 ± 0.787 | 0.0288 ± 0.0018 | 1.000 ± 0.000 | 0.00007 ± 0.00002 | 1.000 ± 0.000 | |
| Vanilla | 0.376 ± 0.016 | 30.977 ± 0.787 | 0.0122 ± 0.0008 | 1.000 ± 0.000 | 0.00001 ± 0.00000 | 1.000 ± 0.000 | |
| FGSM | FGSM | 0.871 ± 0.012 | 30.196 ± 0.449 | 0.0288 ± 0.0001 | 1.000 ± 0.000 | 0.00011 ± 0.00001 | 1.000 ± 0.000 |
| FGSM | 0.751 ± 0.009 | 30.196 ± 0.449 | 0.0249 ± 0.0001 | 1.000 ± 0.000 | 0.00005 ± 0.00001 | 1.000 ± 0.000 | |
| FGSM | 0.289 ± 0.006 | 30.196 ± 0.449 | 0.0096 ± 0.0001 | 1.000 ± 0.000 | 0.00001 ± 0.00000 | 1.000 ± 0.000 | |
| FGSM | 0.949 ± 0.034 | 30.568 ± 0.322 | 0.0310 ± 0.0008 | 1.000 ± 0.000 | 0.00013 ± 0.00002 | 1.000 ± 0.000 | |
| 0.782 ± 0.024 | 30.568 ± 0.322 | 0.0256 ± 0.0005 | 1.000 ± 0.000 | 0.00006 ± 0.00001 | 1.000 ± 0.000 | ||
| 0.328 ± 0.011 | 30.568 ± 0.322 | 0.0107 ± 0.0003 | 1.000 ± 0.000 | 0.00001 ± 0.00000 | 1.000 ± 0.000 | ||
| FGSM | 0.961 ± 0.011 | 30.135 ± 0.340 | 0.0319 ± 0.0000 | 1.000 ± 0.000 | 0.00012 ± 0.00000 | 1.000 ± 0.000 | |
| 0.783 ± 0.008 | 30.135 ± 0.340 | 0.0260 ± 0.0002 | 1.000 ± 0.000 | 0.00006 ± 0.00000 | 1.000 ± 0.000 | ||
| 0.326 ± 0.004 | 30.135 ± 0.340 | 0.0108 ± 0.0002 | 1.000 ± 0.000 | 0.00001 ± 0.00000 | 1.000 ± 0.000 | ||
| Extreme Low Budget | |||||||
| Vanilla | FGSM | 2.262 ± 0.081 | 30.977 ± 0.787 | 0.0731 ± 0.0041 | 1.000 ± 0.000 | 0.00068 ± 0.00010 | 0.918 ± 0.065 |
| Vanilla | 1.768 ± 0.074 | 30.977 ± 0.787 | 0.0571 ± 0.0036 | 1.000 ± 0.000 | 0.00029 ± 0.00006 | 1.000 ± 0.001 | |
| Vanilla | 0.751 ± 0.033 | 30.977 ± 0.787 | 0.0243 ± 0.0015 | 1.000 ± 0.000 | 0.00005 ± 0.00001 | 1.000 ± 0.000 | |
| FGSM | FGSM | 1.545 ± 0.043 | 30.158 ± 0.924 | 0.0512 ± 0.0002 | 1.000 ± 0.000 | 0.00037 ± 0.00004 | 0.999 ± 0.001 |
| FGSM | 1.405 ± 0.039 | 30.158 ± 0.924 | 0.0466 ± 0.0002 | 1.000 ± 0.000 | 0.00018 ± 0.00002 | 1.000 ± 0.000 | |
| FGSM | 0.528 ± 0.016 | 30.158 ± 0.924 | 0.0175 ± 0.0003 | 1.000 ± 0.000 | 0.00003 ± 0.00000 | 1.000 ± 0.000 | |
| FGSM | 1.716 ± 0.033 | 30.483 ± 0.424 | 0.0563 ± 0.0013 | 1.000 ± 0.000 | 0.00044 ± 0.00007 | 0.996 ± 0.003 | |
| 1.454 ± 0.022 | 30.483 ± 0.424 | 0.0477 ± 0.0007 | 1.000 ± 0.000 | 0.00019 ± 0.00003 | 1.000 ± 0.000 | ||
| 0.602 ± 0.009 | 30.483 ± 0.424 | 0.0197 ± 0.0002 | 1.000 ± 0.000 | 0.00003 ± 0.00000 | 1.000 ± 0.000 | ||
| FGSM | 1.792 ± 0.048 | 30.046 ± 0.504 | 0.0596 ± 0.0006 | 1.000 ± 0.000 | 0.00046 ± 0.00003 | 0.994 ± 0.004 | |
| 1.482 ± 0.036 | 30.046 ± 0.504 | 0.0493 ± 0.0005 | 1.000 ± 0.000 | 0.00019 ± 0.00002 | 1.000 ± 0.000 | ||
| 0.619 ± 0.011 | 30.046 ± 0.504 | 0.0206 ± 0.0003 | 1.000 ± 0.000 | 0.00003 ± 0.00000 | 1.000 ± 0.000 | ||
| Medium Budget | |||||||
| Vanilla | FGSM | 4.499 ± 0.158 | 30.977 ± 0.787 | 0.1454 ± 0.0080 | 0.996 ± 0.003 | 0.00263 ± 0.00035 | 0.017 ± 0.012 |
| Vanilla | 3.511 ± 0.147 | 30.977 ± 0.787 | 0.1135 ± 0.0071 | 0.999 ± 0.001 | 0.00112 ± 0.00022 | 0.526 ± 0.198 | |
| Vanilla | 1.493 ± 0.065 | 30.977 ± 0.787 | 0.0482 ± 0.0031 | 1.000 ± 0.000 | 0.00021 ± 0.00004 | 1.000 ± 0.000 | |
| FGSM | FGSM | 2.570 ± 0.056 | 28.917 ± 0.520 | 0.0889 ± 0.0016 | 1.000 ± 0.000 | 0.00105 ± 0.00015 | 0.575 ± 0.140 |
| FGSM | 2.519 ± 0.037 | 28.917 ± 0.520 | 0.0871 ± 0.0016 | 1.000 ± 0.000 | 0.00063 ± 0.00010 | 0.948 ± 0.053 | |
| FGSM | 0.885 ± 0.018 | 28.917 ± 0.520 | 0.0306 ± 0.0007 | 1.000 ± 0.000 | 0.00008 ± 0.00001 | 1.000 ± 0.000 | |
| FGSM | 2.876 ± 0.064 | 28.991 ± 0.314 | 0.0992 ± 0.0014 | 1.000 ± 0.000 | 0.00138 ± 0.00015 | 0.318 ± 0.111 | |
| 2.515 ± 0.059 | 28.991 ± 0.314 | 0.0867 ± 0.0013 | 1.000 ± 0.000 | 0.00064 ± 0.00008 | 0.946 ± 0.040 | ||
| 1.018 ± 0.024 | 28.991 ± 0.314 | 0.0351 ± 0.0005 | 1.000 ± 0.000 | 0.00011 ± 0.00001 | 1.000 ± 0.000 | ||
| FGSM | 3.377 ± 0.093 | 30.431 ± 0.536 | 0.1110 ± 0.0011 | 1.000 ± 0.000 | 0.00154 ± 0.00017 | 0.215 ± 0.029 | |
| 2.839 ± 0.069 | 30.431 ± 0.536 | 0.0933 ± 0.0008 | 1.000 ± 0.000 | 0.00060 ± 0.00009 | 0.959 ± 0.031 | ||
| 1.187 ± 0.026 | 30.431 ± 0.536 | 0.0390 ± 0.0003 | 1.000 ± 0.000 | 0.00011 ± 0.00002 | 1.000 ± 0.000 | ||
| High Budget | |||||||
| Vanilla | FGSM | 8.967 ± 0.299 | 30.977 ± 0.787 | 0.2897 ± 0.0152 | 0.874 ± 0.063 | 0.00964 ± 0.00125 | 0.000 ± 0.000 |
| Vanilla | 6.963 ± 0.285 | 30.977 ± 0.787 | 0.2250 ± 0.0139 | 0.949 ± 0.033 | 0.00410 ± 0.00082 | 0.000 ± 0.001 | |
| Vanilla | 2.965 ± 0.129 | 30.977 ± 0.787 | 0.0958 ± 0.0061 | 1.000 ± 0.001 | 0.00083 ± 0.00013 | 0.788 ± 0.140 | |
| FGSM | FGSM | 3.557 ± 0.194 | 23.117 ± 1.095 | 0.1453 ± 0.0067 | 0.993 ± 0.013 | 0.00320 ± 0.00081 | 0.009 ± 0.014 |
| FGSM | 3.632 ± 0.139 | 23.117 ± 1.095 | 0.1572 ± 0.0057 | 0.986 ± 0.016 | 0.00259 ± 0.00053 | 0.025 ± 0.026 | |
| FGSM | 1.130 ± 0.070 | 23.117 ± 1.095 | 0.0489 ± 0.0027 | 1.000 ± 0.000 | 0.00028 ± 0.00007 | 1.000 ± 0.000 | |
| FGSM | 3.959 ± 0.109 | 23.396 ± 0.141 | 0.1692 ± 0.0049 | 0.994 ± 0.007 | 0.00438 ± 0.00077 | 0.001 ± 0.001 | |
| 3.618 ± 0.070 | 23.396 ± 0.141 | 0.1546 ± 0.0033 | 0.989 ± 0.013 | 0.00242 ± 0.00045 | 0.033 ± 0.041 | ||
| 1.438 ± 0.036 | 23.396 ± 0.141 | 0.0615 ± 0.0016 | 1.000 ± 0.000 | 0.00043 ± 0.00009 | 0.998 ± 0.003 | ||
| FGSM | 6.088 ± 0.209 | 29.736 ± 0.814 | 0.2047 ± 0.0030 | 0.992 ± 0.003 | 0.00574 ± 0.00064 | 0.000 ± 0.000 | |
| 5.210 ± 0.171 | 29.736 ± 0.814 | 0.1752 ± 0.0026 | 0.994 ± 0.001 | 0.00223 ± 0.00024 | 0.040 ± 0.037 | ||
| 2.137 ± 0.065 | 29.736 ± 0.814 | 0.0719 ± 0.0015 | 1.000 ± 0.000 | 0.00041 ± 0.00004 | 0.997 ± 0.003 | ||
| Extreme High Budget | |||||||
| Vanilla | FGSM | 17.525 ± 0.487 | 30.977 ± 0.787 | 0.5661 ± 0.0247 | 0.240 ± 0.030 | 0.03412 ± 0.00457 | 0.000 ± 0.000 |
| Vanilla | 13.589 ± 0.501 | 30.977 ± 0.787 | 0.4390 ± 0.0250 | 0.508 ± 0.052 | 0.01521 ± 0.00349 | 0.000 ± 0.000 | |
| Vanilla | 5.885 ± 0.253 | 30.977 ± 0.787 | 0.1902 ± 0.0119 | 0.983 ± 0.010 | 0.00308 ± 0.00049 | 0.006 ± 0.006 | |
| FGSM | FGSM | 3.152 ± 0.182 | 13.333 ± 0.688 | 0.2364 ± 0.0068 | 0.945 ± 0.032 | 0.00913 ± 0.00132 | 0.000 ± 0.000 |
| FGSM | 3.887 ± 0.169 | 13.333 ± 0.688 | 0.2916 ± 0.0027 | 0.804 ± 0.022 | 0.00891 ± 0.00094 | 0.000 ± 0.000 | |
| FGSM | 1.108 ± 0.071 | 13.333 ± 0.688 | 0.0831 ± 0.0026 | 1.000 ± 0.000 | 0.00091 ± 0.00011 | 0.711 ± 0.111 | |
| FGSM | 4.077 ± 0.069 | 14.473 ± 0.350 | 0.2817 ± 0.0023 | 0.887 ± 0.014 | 0.01461 ± 0.00072 | 0.000 ± 0.000 | |
| 3.719 ± 0.046 | 14.473 ± 0.350 | 0.2570 ± 0.0035 | 0.908 ± 0.003 | 0.00717 ± 0.00039 | 0.000 ± 0.000 | ||
| 1.576 ± 0.011 | 14.473 ± 0.350 | 0.1089 ± 0.0019 | 0.995 ± 0.006 | 0.00152 ± 0.00010 | 0.214 ± 0.055 | ||
| FGSM | 9.052 ± 0.403 | 25.573 ± 1.730 | 0.3543 ± 0.0084 | 0.762 ± 0.036 | 0.01708 ± 0.00081 | 0.000 ± 0.000 | |
| 7.970 ± 0.324 | 25.573 ± 1.730 | 0.3120 ± 0.0082 | 0.858 ± 0.038 | 0.00647 ± 0.00070 | 0.000 ± 0.000 | ||
| 3.203 ± 0.120 | 25.573 ± 1.730 | 0.1254 ± 0.0040 | 0.997 ± 0.004 | 0.00143 ± 0.00017 | 0.286 ± 0.117 |
Table 5.
| Trained | Attack | Mean Disp. | Median Clean–Clean | Norm. Disp. | kNN Rec. | MMD | |
| Extreme Low Budget | |||||||
| Vanilla | FGSM | 0.00169 ± 0.00028 | 0.02985 ± 0.00033 | 0.0565 ± 0.0088 | 0.899 ± 0.037 | 0.00125 ± 0.00076 | 0.432 ± 0.405 |
| Vanilla | 0.00150 ± 0.00033 | 0.02980 ± 0.00031 | 0.0503 ± 0.0105 | 0.912 ± 0.039 | 0.00126 ± 0.00099 | 0.480 ± 0.448 | |
| Vanilla | 0.00097 ± 0.00027 | 0.02969 ± 0.00034 | 0.0328 ± 0.0087 | 0.921 ± 0.026 | 0.00087 ± 0.00059 | 0.588 ± 0.365 | |
| FGSM | FGSM | 0.00113 ± 0.00015 | 0.02971 ± 0.00026 | 0.0381 ± 0.0050 | 0.953 ± 0.018 | 0.00040 ± 0.00023 | 0.891 ± 0.139 |
| FGSM | 0.00119 ± 0.00020 | 0.02969 ± 0.00025 | 0.0400 ± 0.0067 | 0.936 ± 0.014 | 0.00059 ± 0.00042 | 0.755 ± 0.305 | |
| FGSM | 0.00076 ± 0.00023 | 0.02950 ± 0.00017 | 0.0259 ± 0.0077 | 0.936 ± 0.029 | 0.00057 ± 0.00042 | 0.736 ± 0.232 | |
| FGSM | 0.00129 ± 0.00023 | 0.02981 ± 0.00084 | 0.0434 ± 0.0068 | 0.945 ± 0.028 | 0.00066 ± 0.00041 | 0.723 ± 0.270 | |
| 0.00116 ± 0.00021 | 0.02977 ± 0.00080 | 0.0389 ± 0.0062 | 0.946 ± 0.027 | 0.00053 ± 0.00042 | 0.804 ± 0.278 | ||
| 0.00080 ± 0.00027 | 0.02965 ± 0.00079 | 0.0271 ± 0.0090 | 0.939 ± 0.027 | 0.00063 ± 0.00047 | 0.712 ± 0.249 | ||
| FGSM | 0.00128 ± 0.00011 | 0.03018 ± 0.00011 | 0.0425 ± 0.0036 | 0.939 ± 0.020 | 0.00047 ± 0.00014 | 0.852 ± 0.105 | |
| 0.00117 ± 0.00017 | 0.03015 ± 0.00014 | 0.0388 ± 0.0054 | 0.941 ± 0.031 | 0.00049 ± 0.00027 | 0.809 ± 0.188 | ||
| 0.00056 ± 0.00009 | 0.03000 ± 0.00019 | 0.0187 ± 0.0029 | 0.970 ± 0.024 | 0.00013 ± 0.00007 | 0.997 ± 0.004 | ||
| Low Budget | |||||||
| Vanilla | FGSM | 0.00291 ± 0.00033 | 0.03005 ± 0.00046 | 0.0966 ± 0.0096 | 0.783 ± 0.015 | 0.00270 ± 0.00135 | 0.123 ± 0.178 |
| Vanilla | 0.00256 ± 0.00033 | 0.02997 ± 0.00037 | 0.0853 ± 0.0100 | 0.817 ± 0.029 | 0.00262 ± 0.00140 | 0.148 ± 0.213 | |
| Vanilla | 0.00141 ± 0.00030 | 0.02980 ± 0.00032 | 0.0472 ± 0.0096 | 0.913 ± 0.031 | 0.00126 ± 0.00084 | 0.438 ± 0.461 | |
| FGSM | FGSM | 0.00192 ± 0.00019 | 0.03000 ± 0.00047 | 0.0640 ± 0.0056 | 0.913 ± 0.030 | 0.00096 ± 0.00041 | 0.531 ± 0.278 |
| FGSM | 0.00183 ± 0.00021 | 0.02997 ± 0.00048 | 0.0609 ± 0.0060 | 0.916 ± 0.036 | 0.00088 ± 0.00042 | 0.573 ± 0.300 | |
| FGSM | 0.00081 ± 0.00014 | 0.02971 ± 0.00046 | 0.0271 ± 0.0044 | 0.971 ± 0.035 | 0.00025 ± 0.00017 | 0.963 ± 0.055 | |
| FGSM | 0.00217 ± 0.00027 | 0.03002 ± 0.00083 | 0.0723 ± 0.0073 | 0.875 ± 0.032 | 0.00145 ± 0.00048 | 0.283 ± 0.154 | |
| 0.00192 ± 0.00018 | 0.02995 ± 0.00081 | 0.0640 ± 0.0045 | 0.902 ± 0.030 | 0.00105 ± 0.00016 | 0.477 ± 0.099 | ||
| 0.00098 ± 0.00009 | 0.02970 ± 0.00076 | 0.0328 ± 0.0022 | 0.952 ± 0.017 | 0.00037 ± 0.00006 | 0.929 ± 0.032 | ||
| FGSM | 0.00210 ± 0.00003 | 0.03049 ± 0.00004 | 0.0716 ± 0.0099 | 0.875 ± 0.008 | 0.00113 ± 0.00010 | 0.408 ± 0.062 | |
| 0.00194 ± 0.00013 | 0.03041 ± 0.00006 | 0.0637 ± 0.0040 | 0.906 ± 0.016 | 0.00088 ± 0.00024 | 0.542 ± 0.165 | ||
| 0.00088 ± 0.00012 | 0.03010 ± 0.00004 | 0.0292 ± 0.0040 | 0.976 ± 0.034 | 0.00023 ± 0.00011 | 0.974 ± 0.035 | ||
| Medium Budget | |||||||
| Vanilla | FGSM | 0.00492 ± 0.00039 | 0.03046 ± 0.00065 | 0.1613 ± 0.0096 | 0.523 ± 0.005 | 0.00606 ± 0.00242 | 0.005 ± 0.008 |
| Vanilla | 0.00423 ± 0.00043 | 0.03023 ± 0.00052 | 0.1397 ± 0.0117 | 0.615 ± 0.013 | 0.00507 ± 0.00131 | 0.010 ± 0.017 | |
| Vanilla | 0.00236 ± 0.00029 | 0.02998 ± 0.00037 | 0.0786 ± 0.0087 | 0.855 ± 0.028 | 0.00259 ± 0.00124 | 0.132 ± 0.194 | |
| FGSM | FGSM | 0.00307 ± 0.00112 | 0.03028 ± 0.00032 | 0.1015 ± 0.0050 | 0.795 ± 0.027 | 0.00202 ± 0.00113 | 0.112 ± 0.018 |
| FGSM | 0.00303 ± 0.00016 | 0.03022 ± 0.00027 | 0.1001 ± 0.0059 | 0.779 ± 0.031 | 0.00180 ± 0.00028 | 0.165 ± 0.063 | |
| FGSM | 0.00125 ± 0.00111 | 0.02988 ± 0.00027 | 0.0418 ± 0.0040 | 0.959 ± 0.040 | 0.0045 ± 0.0012 | 0.876 ± 0.083 | |
| FGSM | 0.00364 ± 0.00038 | 0.03062 ± 0.00029 | 0.1189 ± 0.0129 | 0.705 ± 0.026 | 0.00368 ± 0.00141 | 0.020 ± 0.020 | |
| 0.00320 ± 0.00036 | 0.03039 ± 0.00032 | 0.1053 ± 0.0124 | 0.751 ± 0.034 | 0.00240 ± 0.00102 | 0.121 ± 0.111 | ||
| 0.00159 ± 0.00035 | 0.03001 ± 0.00021 | 0.0529 ± 0.0120 | 0.942 ± 0.036 | 0.00096 ± 0.00078 | 0.592 ± 0.413 | ||
| FGSM | 0.00380 ± 0.00008 | 0.03086 ± 0.00020 | 0.1231 ± 0.0023 | 0.671 ± 0.003 | 0.00315 ± 0.00088 | 0.018 ± 0.002 | |
| 0.00338 ± 0.00018 | 0.03070 ± 0.00023 | 0.1101 ± 0.0053 | 0.720 ± 0.020 | 0.00233 ± 0.00046 | 0.072 ± 0.053 | ||
| 0.00161 ± 0.00014 | 0.03025 ± 0.00024 | 0.0532 ± 0.0045 | 0.927 ± 0.041 | 0.00069 ± 0.00020 | 0.703 ± 0.156 | ||
| High Budget | |||||||
| Vanilla | FGSM | 0.00816 ± 0.00026 | 0.03087 ± 0.00048 | 0.2644 ± 0.0047 | 0.247 ± 0.025 | 0.01396 ± 0.00250 | 0.000 ± 0.000 |
| Vanilla | 0.00692 ± 0.00037 | 0.03035 ± 0.00061 | 0.2278 ± 0.0081 | 0.348 ± 0.020 | 0.01156 ± 0.00179 | 0.000 ± 0.000 | |
| Vanilla | 0.00382 ± 0.00033 | 0.03022 ± 0.00055 | 0.1262 ± 0.0086 | 0.665 ± 0.019 | 0.00461 ± 0.00182 | 0.014 ± 0.023 | |
| FGSM | FGSM | 0.00436 ± 0.00005 | 0.03040 ± 0.00031 | 0.1436 ± 0.0030 | 0.562 ± 0.019 | 0.00400 ± 0.0023 | 0.008 ± 0.003 |
| FGSM | 0.00441 ± 0.00017 | 0.03040 ± 0.00027 | 0.1451 ± 0.0069 | 0.540 ± 0.038 | 0.00351 ± 0.00087 | 0.015 ± 0.011 | |
| FGSM | 0.00173 ± 0.00011 | 0.02976 ± 0.00020 | 0.0582 ± 0.0041 | 0.948 ± 0.015 | 0.00082 ± 0.00023 | 0.601 ± 0.164 | |
| FGSM | 0.00505 ± 0.00042 | 0.03126 ± 0.00048 | 0.1615 ± 0.0119 | 0.477 ± 0.063 | 0.00503 ± 0.00143 | 0.003 ± 0.004 | |
| 0.00458 ± 0.00037 | 0.03102 ± 0.00052 | 0.1477 ± 0.0103 | 0.504 ± 0.066 | 0.00381 ± 0.00158 | 0.035 ± 0.057 | ||
| 0.00221 ± 0.00017 | 0.03036 ± 0.00051 | 0.0727 ± 0.0042 | 0.883 ± 0.034 | 0.00138 ± 0.00039 | 0.312 ± 0.149 | ||
| FGSM | 0.00601 ± 0.00016 | 0.03102 ± 0.00042 | 0.1937 ± 0.0034 | 0.383 ± 0.042 | 0.00619 ± 0.00107 | 0.000 ± 0.001 | |
| 0.00536 ± 0.00011 | 0.03069 ± 0.00037 | 0.1746 ± 0.0141 | 0.436 ± 0.039 | 0.00420 ± 0.00041 | 0.003 ± 0.002 | ||
| 0.00272 ± 0.00008 | 0.03042 ± 0.00030 | 0.0894 ± 0.0026 | 0.837 ± 0.019 | 0.00160 ± 0.00024 | 0.218 ± 0.080 | ||
| Extreme High Budget | |||||||
| Vanilla | FGSM | 0.01251 ± 0.00033 | 0.03022 ± 0.00044 | 0.4140 ± 0.0144 | 0.108 ± 0.012 | 0.02170 ± 0.00072 | 0.000 ± 0.000 |
| Vanilla | 0.01082 ± 0.00014 | 0.03021 ± 0.00050 | 0.3583 ± 0.0080 | 0.160 ± 0.025 | 0.02063 ± 0.00063 | 0.000 ± 0.000 | |
| Vanilla | 0.00616 ± 0.00032 | 0.03042 ± 0.00054 | 0.2024 ± 0.0070 | 0.399 ± 0.011 | 0.00901 ± 0.00233 | 0.000 ± 0.000 | |
| FGSM | FGSM | 0.00613 ± 0.00009 | 0.02984 ± 0.00061 | 0.2056 ± 0.0034 | 0.371 ± 0.016 | 0.00576 ± 0.00055 | 0.001 ± 0.001 |
| FGSM | 0.00745 ± 0.00025 | 0.03042 ± 0.00057 | 0.2450 ± 0.0038 | 0.291 ± 0.031 | 0.00758 ± 0.00177 | 0.000 ± 0.000 | |
| FGSM | 0.00248 ± 0.00006 | 0.02978 ± 0.00046 | 0.0832 ± 0.0023 | 0.860 ± 0.018 | 0.00136 ± 0.00014 | 0.284 ± 0.061 | |
| FGSM | 0.00753 ± 0.00019 | 0.03220 ± 0.00039 | 0.2338 ± 0.0075 | 0.225 ± 0.013 | 0.01125 ± 0.00254 | 0.000 ± 0.000 | |
| 0.00672 ± 0.00021 | 0.03197 ± 0.00038 | 0.2132 ± 0.0085 | 0.252 ± 0.018 | 0.00758 ± 0.00198 | 0.000 ± 0.001 | ||
| 0.00325 ± 0.00030 | 0.03110 ± 0.00024 | 0.1045 ± 0.0091 | 0.658 ± 0.021 | 0.00236 ± 0.00045 | 0.078 ± 0.055 | ||
| FGSM | 0.00952 ± 0.00012 | 0.03171 ± 0.00042 | 0.3004 ± 0.0042 | 0.138 ± 0.025 | 0.02139 ± 0.00051 | 0.000 ± 0.000 | |
| 0.00833 ± 0.00002 | 0.03145 ± 0.00017 | 0.2648 ± 0.0019 | 0.164 ± 0.021 | 0.01402 ± 0.00193 | 0.000 ± 0.000 | ||
| 0.00410 ± 0.00038 | 0.03075 ± 0.00025 | 0.1333 ± 0.0112 | 0.597 ± 0.039 | 0.00363 ± 0.00160 | 0.030 ± 0.039 |
Table 6. Geometry in the 3-component M-PHATE embedding of the joint clean/perturbed tensor. Columns match Table 5. Mean ± s.d. over seeds 0, 1, 2.

Figure 8. Embeddings under the attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).

Figure 9. Embeddings under the attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).
Computing resources
All experiments were run on a single machine: one NVIDIA Tesla V100 (16 GB), a 16-core x86_64 CPU, and 64 GB of system memory.
Each of the five budget sets is one self-contained run covering 3 seeds 4 networks: 12 networks trained for 50 epochs. A run took roughly 20 minutes, and the five sets together took 101.9 minutes of wall-clock time. Network training is a minority of that cost: 26.2 minutes in total, against 35.0 minutes for the embeddings and 39.4 minutes for the permutation tests (0.9 minutes for the kNN and distance computations). Embedding, not training, dominates this pipeline, and the cost depends strongly on the number of slices. Averaged over exploratory runs, a 50-slice training-dynamics embedding takes 489 60 s, while a 2-slice clean-versus-perturbed embedding takes 124 30 s.
References
- [1]Christian Szegedy, Wojciech Zaremba, I. Sutskever, Joan Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. CoRR, December 2013.arxiv.org/abs/1312.6199
- [2]Ian J. Goodfellow, Jonathan Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. URL https://arxiv.org/abs/1412.6572.
- [3]Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks, September 2019.
- [4]Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness May Be at Odds with Accuracy, September 2019.arxiv.org/abs/1805.12152
- [5]Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019.arxiv.org/abs/1901.08573
- [6]Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In H. Wallace, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. 2019.arxiv.org/abs/1905.02175
- [7]Liwei Wang, Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hopcroft. Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation, November 2018.arxiv.org/abs/1810.11750
- [8]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. In Proceedings of the 36th International Conference on Machine Learning, pages 3519–3529, May 2019.arxiv.org/abs/1905.00414
- [9]Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.05806
- [10]Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25):723–773, 2012. ISSN 1533-7928.DOI
- [11]Scott Gigante, Adam S. Charles, Smita Krishnaswamy, and Gal Mishne. Visualizing the PHATE of Neural Networks. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1908.02831
- [12]Kevin R. Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B. Burkhardt, William S. Chen, Kristina Yim, Antonia van den Elzen, Matthew J. Hirn, Ronald R. Coifman, Natalia B. Ivanova, Guy Wolf, and Smita Krishnaswamy. Visualizing structure and transitions in high-dimensional biological data. Nature Biotechnology, 37(12):1482–1492, December 2019. ISSN 1546-1696. doi: 10.1038/s41587-019-0336-3.DOI
- [13]Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial Robustness as a Prior for Learned Representations, September 2019.arxiv.org/abs/1906.00945
- [14]Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting Neural Network Accuracy from Weights, April 2021.arxiv.org/abs/2002.11448
- [15]Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the Classifier: Dissecting the Weight Space of Neural Networks. In ECAI 2020, pages 1119–1126. 2020. doi: 10.3233/FAIA200209.DOI
- [16]Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i-Nieto, and Damian Borth. Model Zoos: A Dataset of Diverse Populations of Neural Network Models. Advances in Neural Information Processing Systems, 35:38134–38148, December 2022. doi: 10.52202/068431-2763.DOI
- [17]Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant Architectures for Learning in Deep Weight Spaces. In Proceedings of the 40th International Conference on Machine Learning, pages 25790–25816, July 2023.arxiv.org/abs/2301.12780
- [18]Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998. ISSN 1558-2256. doi: 10.1109/5.726791.
- [19]Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020.arxiv.org/abs/2003.01690
- [20]Jiancheng Xie, Lou C Voinov, Noga Mudrik, Gal Mishne, and Adam Shabti Charles. Multiway multislice PHATE: Visualizing hidden dynamics of RNNs through training. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https://openreview.net/forum?id=9Yr4V7iZsq.
- [21]Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BJxO4OEFvH.arxiv.org/abs/2001.03994
- [22]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015.arxiv.org/abs/1512.03385
- [23]Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Proceedings of the British Machine Vision Conference 2016, pages 87.1–87.12, York, UK, 2016. ISBN 978-1-901725-59-9. doi: 10.5244/C.30.87.DOI
- [24]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021.