WorksComputer Science
The geometry of neural network decision boundaries
Abstract
Artificial neural networks (ANNs) are built using well-understood mathematical operations, yet their complex nature poses significant challenges to intuitively understanding their decision-making processes. This work introduces a method for visualizing and measuring ANN decision boundary geometry directly in the high-dimensional input space, avoiding distortions found with traditional projection-based techniques. We analyze boundary curvatures and distances in diverse inputs, architectures, and training methods. Boundary distances grow smoothly and slowly with consistent curvature as the size of the subspace increases, suggesting that decision boundaries are close to input samples within large, continuous subspaces. We also quantify how much boundary distances and curvatures reduce near dataset samples after adversarial training. Training methodology has a significantly stronger impact on boundary distance and curvature than architecture or test-set accuracy. These findings are contextualized within prior work, supporting or refuting several previously proposed theories of boundary behavior. Our analytically and empirically supported methodology lays the foundation for future studies to improve network interpretability and understanding.
Introduction
An artificial neural network (ANN) used for classification can be described entirely in terms of its category decision boundaries in the input space. Understanding these boundaries is crucial for enhancing robustness and interpretability, particularly for applications in high-stakes domains. However, measuring or visualizing those boundaries is challenging due to the high-dimensional and non-linear relationship between the network’s inputs and outputs. A common approach to understanding these boundaries is to visualize them by projecting the decision surface onto a two-dimensional plane [1, 2]. Although informative, this suffers from limitations: high-dimensional points can map to the same projected location, the distances between points are distorted, and the projections lack invertibility back to the original high-dimensional decision space [3, 4]. An alternative strategy investigates the decision boundary directly in the high-dimensional input space. Early work in this direction [5 Baehrens] et al., 2010] laid the foundation for the modern focus on adversarial attacks [7, 8]. Examples produced by adversarial attacks are a worst-case demonstration of a classifier’s inability to gracefully cope with identity-preserving shifts or distortions of its input [9]. To construct one, an adversary perturbs the ANN input in a small but specific way such that the output changes significantly. The perturbation is optimized to be as close as possible to the ANN’s decision boundary.
Adversarial examples provide context for the ultimate goal of improving interpretability and our understanding of classifier behavior by measuring and visualizing the output space near unaltered dataset samples. Previous work has explored ANN responses in the proximity of adversarial examples [10, 11, 12, 13, 14, 15], or within hand-picked two-dimensional input domains [16]. However, these methods analyze one or two boundary dimensions at a time, leaving key questions unanswered about the high-dimensional structure of the boundary. For example: How many orthogonal directions from an input sample maintain close proximity to the decision boundary? How does the boundary behave between these directions? Does the boundary distance increase smoothly or does it fluctuate with increased search space dimensionality? What is the high-dimensional distribution of class label regions around dataset inputs?
In this work, we introduce a novel approach that quantitatively analyzes the geometry of ANN decision boundaries in high-dimensional subspaces centered on dataset samples. Our method provides the first evidence that decision boundaries are close to natural inputs within large subspaces across architectures and datasets, supporting the hypothesis that adversarial examples occupy dense regions of input space [17, 11, 18, 12]. Specifically, we propose an untargeted adversarial attack framework coupled with a principled method to measure high-dimensional boundary curvature. Our approach corrects inaccuracies in prior boundary curvature estimates and enables more precise 2D visualizations.
Our findings reveal a large subspace around input samples where the decision boundary remains only marginally farther away than at its closest point. Moreover, we show that boundary distances increase smoothly as the dimensionality of the subspace grows. We further demonstrate that adversarial training, widely considered the state-of-the-art defense against adversarial attacks [19, 20], not only increases boundary distance but also reduces boundary curvature, supporting prior hypotheses [21]. Additionally, we provide new evidence that adversarial training affects class label composition near natural inputs, indicating reduced semantic information in label proximity.
In summary, we contribute the first quantitative analysis of the geometry of nonlinear ANN decision boundaries in high-dimensional subspaces centered on dataset samples, a novel curvature measurement method that corrects errors in prior approaches, and new insights into adversarial training’s impact on boundary smoothness and label distributions. Our experiments confirm consistent trends across datasets, architectures, and generalization performance, offering a principled foundation for understanding decision boundaries and improving interpretability in deep learning.
Methods
We aim to describe both the proximity and the curvature of the decision boundary within high-dimensional subspaces centered on test-set inputs. To demonstrate the robustness of our findings, we compare two ResNet architectures [22, 19] trained on the CIFAR-10 dataset [23], which differ in their training parameters and generalization performance. We then perform a more detailed curvature analysis on the smaller of the two architectures. Additional dataset and model details are provided in Appendix B. Control experiments that validate consistency across datasets and random seeds are provided in Appendices C, D, and E. Our experiments were conducted on 100 test images, chosen to ensure equal label representation (10 images per label), all of which were correctly classified by the networks under study.
Defining subspaces with minimum boundary distance
Consider a neural network that receives an -pixel input, , and produces an output vector of normalized scores associated with label categories, . We define a subspace centered on the input using a set of orthogonal vectors , . The method for finding this set must jointly enforce a bounded input, , orthogonality, , , and misclassification , , where is the correct class index. Additionally, the perturbation vector should have minimal length, , where is the norm. While solving this exactly for deep networks is infeasible, it can be approximated using an untargeted adversarial attack objective. We define , which includes the cross-entropy loss
Here, is a penalty coefficient optimized via binary search at run time. The components combine to give the full objective function as implemented:
which we solved using the interior-point optimizer implemented by Ipopt [24]. This is in line with prior work that uses the cross-entropy loss in an adversarial objective [8, 25], but modified to match an untargeted attack objective [11, 26].
After iteratively minimizing this objective, the first vector points to the closest decision boundary, while subsequent vectors correspond to orthogonal directions with increasing distances to the boundary. The number of vectors is limited by computational constraints, but the method can theoretically scale up to the input dimensionality. This greedy approach provides a subspace that is constrained by an empirical upper bound for boundary distances.
We used sampling to find boundary distances at locations between subspace axes. Specifically, within each subspace, we sampled 100 random directions using a linear combination of normally distributed variables, a method known to provide even sampling in high-dimensional spaces [28]. We then used a binary search to find the decision boundary along each sampled direction. All distances are measured using the -norm. Improved methods for finding orthogonal adversarial perturbations, sampling, or searching for the boundary could further refine our measurements. However, we do not expect such an improvement to qualitatively modify our conclusions.
Measuring decision boundary curvature
Neural network decision boundaries can be sufficiently described as high-dimensional Riemannian manifolds. Building on this perspective, we derive a formula for quantifying the curvature of a decision boundary both in its native dimensionality as well as within any subspace containing the boundary.
Let be the functions corresponding to the -th and -th logits of a neural network, . Define . Then the -decision boundary is
The set is independent of the sign of (i.e., ), though the ordering will affect the sign of curvature. We follow the convention established earlier and in Moosavi-Dezfooli et al. [ [21] by associating with the correct class label for a point and with an incorrect label. This ensures that positive curvature of near corresponds to the boundary bowing inward towards , while negative curvature indicates the boundary bowing outward (see Figure 1 for an example of positive curvature). If the curvature is zero near , then is locally affine, or flat.
Figure 1. The decision boundary for an adversarially trained ResNet-50. (Left) A subspace centered on a dataset image (class 5 exemplar, bottom middle) is defined, with each orthogonal axis corresponding to a direction where the decision boundary is closest. Principal curvature is measured at random points within the subspace. (Right) A 2-dimensional subspace is visualized using two adversarial directions, accurately showing the boundary shape and distances to alternate class labels. Perturbation images outside the graph are rescaled for visualization, with their length indicated on the axes. Each color represents a class (e.g., deer: class 4, dogs: class 5, horses: class 7). White arrows indicate sampling directions within the subspace to locate the boundary.
We apply principal curvature decomposition at 50 boundary points to quantify the extent of linearity. This decomposition, performed on a level-set of the decision space (generally of codimension 1), yields scalars and vectors called the principal curvatures and principal curvature directions, respectively. Conceptually analogous to principal component analysis, which organizes the space as an orthogonal basis of directions of maximal variance, principal curvature analysis organizes the (tangential) space of the decision boundary into orthogonal directions of maximum curvature. Specifically, the principal curvatures and associated directions correspond to the eigenvectors and eigenvalues of a linear operator called the shape operator.
To investigate curvature within a subspace, we define the subspace pullback of the shape operator:
where is an orthogonal projection matrix, and the image of spans the subspace of . This formulation restricts the shape operator to a linear subspace of the tangential space. It also allows for a direct comparison of curvature between a subspace and the full response space. The mathematical motivation for this definition is rooted in differential geometry, with a full treatment provided in Appendix § 8.
Results
The distance to the decision boundary increases with each additional dimension of our defined subspace. The term dimension number refers to the ordered axes for the subspace, where the lowest dimension number corresponds to the first axis with the closest boundary point, and higher dimension numbers indicate increasingly distant boundary points. The perturbations become visually noticeable at approximately dimension number 20 for the robust network and beyond dimension number 50 for the naturally trained network (see Appendix § 10 Figure 6 for examples).
Boundary distances
The circles in the top left panel of Figure 2 represent the average boundary distances along each dimension number, computed across test images by greedily minimizing Equation 2.1. The box plots in the top-left panel of Figure 2 display the distribution of decision boundary distances across sampled directions and test images. These results confirm that the mean initial perturbation distances align with those reported in prior work on adversarial attacks [29] and decision boundary distance estimations [30] (also see Appendix Table 2). The bottom-left panel of Figure 2 shows the distribution of distance coefficients, defined as the ratio of the maximum distance to the decision boundary within a subspace to the largest boundary distance along the defining subspace axes (which will be along the axis with the highest dimension number). For example, a value of 2 indicates that the farthest sampled boundary point is twice as distant as the farthest boundary point along an axis. Histograms in the right-hand panels of Figure 2 count sampled boundary distances, with subspace dimensionality increasing from left to right and top to bottom. They indicate that the decision boundary distances increase smoothly across dimensions, with minimal variation between subspace axes. Distances were only reported for boundary points that remained within the pixel bounds, covering at least 40% of the sampled directions for the naturally trained network and 60% for the adversarially trained network (see Appendix § 10 Figure 7).
Figure 2. Boundary distances. The distance from a test set image to the decision boundary grows sub-linearly with increasing subspace dimensionality and remains consistent across model architectures. (Left, top) Box plots show decision boundary distances for uniformly sampled directions within n-dimensional subspaces (horizontal axis), with dots indicating distances along the subspace axes, averaged across images. Boxes show interquartile ranges, whiskers represent the 10% and 90% percentiles, and dashed lines show random sample means. Every other dimension up to 50 is shown. (Left, bottom) Similar to the top panel, but the vertical axis shows the largest boundary distance within the subspace divided by the largest boundary distance along the subspace axes. Low values indicate that the boundary remains close to the input sample between the axes. (Right) Histograms of boundary distances for randomly sampled directions within subspaces, with dimensionality increasing across subplots (odd-numbered dimensions are shown). The mean shifts rightward and variance increases proportionately with dimensionality, suggesting a smooth growth in distance with increasing dimensionality. Line colors represent training schemes as per the legend.
Figure 7. Ratio of out of bounds samples. We uniformly sample the distance to the decision boundary with increasing dimensionality per input image and adversarial subspace dimension. We report the fraction of directions pointing to out-of-bounds boundaries, averaged across 100 images.
The distance to the decision boundary grows sub-linearly as the dimensionality of the subspace increases, with the most significant growth occurring in the first few dimensions. The adversarially trained network consistently exhibits smaller distance coefficients, suggesting less variability in boundary distances within the subspace.
Boundary visualizations
Figure 3 presents two-dimensional cross-sections of the decision surface. The horizontal axis in each cross-section corresponds to a specific subspace axis, with the associated dimension number labeled. The black arrow marks the farthest decision boundary point sampled within the subspace. The vertical axis is determined through one step of the Gram-Schmidt process, ensuring that the visualized space is spanned by the horizontal axis and the vector pointing to the farthest sampled boundary point.
Figure 3. Boundary visualizations. Each subplot shows a two-dimensional cross-section of the decision space for a naturally trained network, centered on a random test set image. The cross-section corresponds to the subspace with the largest distance coefficient (top-right corner), calculated as the ratio of the largest boundary distance within the subspace (black arrow) to the largest adversarial perturbation defining the subspace (horizontal axis, with dimension number indicated). Red dots mark the described adversarial perturbations. Shaded areas represent regions outside the image bounds. Unlike projections, this visualization preserves high-dimensional spatial relationships, ensuring accurate representation of boundary geometry.
The cross-sections are ordered by their distance coefficient, displayed in the upper-right corner of each plot. These visualizations include instances where the decision boundary distance increases smoothly between axes, as well as less frequent cases where the boundary extends significantly farther from the origin between the subspace axes (a pattern also observed in Figure 2).
Boundary curvature
Figure 4 presents boundary curvature measurements under various experimental conditions, including different training methods, boundary locations, and subspaces. Panels A and B display the mean curvature (i.e. the mean of the eigenvalues of the shape operator) for points on the decision boundary. Boundary points separating unperturbed test set examples from adversarial points (adversarial boundary; Panel B) exhibit higher mean curvature compared to boundary points separating the examples themselves (test boundary; Panel A). Panel C shows a scatter plot of the principal curvatures (i.e. eigenvalues of the shape operator) at the found boundary points in the high-dimensional input space. Both positive and negative curvature values are observed at all boundary points. Panels D and E report principal curvatures within subspaces defined by random vectors and by adversarial vectors, respectively. Curvature within the found subspaces (Panel E) is notably larger than in random subspaces (Panel D). For naturally trained networks, the boundary points in both the full input space (Panel B) and the found subspace (Panel E) tend to exhibit more negative mean curvature. The subspace curvature is measured using the pullback defined in Equation (14), which establishes bounds on curvature amounts between each principal vector. Boundary points closest to the clean input can exhibit positive curvature only if the radius of curvature exceeds the distance to the boundary.

Figure 4. Boundary curvature. Curvature data for 50 correctly classified test images, with evenly distributed labels for the first 10 dimension numbers. (A, B) Mean curvatures per boundary point are smaller for boundaries separating two test images (“test boundary”) compared to those separating a test image and an adversarial image (“adversarial boundary”). (C) Scatter plot of curvature profiles for all images and directions under the adversarial boundary condition. The red inset shows the full dimensionality (same horizontal axis as the main plot) with a zoomed vertical axis, highlighting that adversarially trained boundaries are flatter (more linear). The data includes all subspace boundary points per test image. (D, E) Curvature of adversarial boundaries within random subspaces (D) and adversarial subspaces (E). Adversarial subspaces exhibit more extreme curvature. Points show the mean across test images; error bars represent standard deviation.
We observe consistent curvature measurements when varying model architectures, test-set accuracy, and random seeds for two different datasets (see Appendices D and E). However, differences in training methodology lead to notable variations in boundary geometry for otherwise identical architectures. Adversarial training linearizes the decision boundary, both between correctly labeled test images and near adversarial examples. The difference in curvature values between naturally and adversarially trained networks is less pronounced in random subspaces than subspaces found via an adversarial objective.
Decision class composition
Figure 5 presents measurements of the diversity of adversarial classes near test samples. The leftmost subplot shows the average number of unique adversarial classes found within the first 50 dimensions of the subspace, computed across correctly classified test images. The remaining subplots display the distribution of adversarial classes within the subspaces, categorized by the original label of the test samples. Table 1 provides the discrete entropy of each distribution, quantifying the diversity of adversarial classes present. Adversarially trained networks have higher entropy values and a broader distribution of adversarial classes compared to naturally trained networks. This suggests that adversarial training leads to an increase in the number of unique adversarial classes near test samples.
Figure 5. Adversarial Class Composition. (Left) Within the subspaces of randomly chosen test images we quantified how many unique adversarial classes exist. In most cases adversarial training increases the diversity of alternate classes close to any test image. (Right) Each plot shows the distribution of nearby alternate classes for each origin class with natural and adversarial training. Adversarial training results in a more even distribution of nearby classes for all categories except bird and ship.
| Label | Natural | Robust |
| airplane | 1.70 | 1.81 |
| car | 1.01 | 1.26 |
| bird | 1.72 | 1.59 |
| cat | 1.56 | 1.97 |
| deer | 1.41 | 1.98 |
| dog | 1.51 | 1.82 |
| frog | 1.12 | 1.51 |
| horse | 1.63 | 1.78 |
| ship | 1.74 | 1.39 |
| truck | 1.79 | 1.88 |
Table 1. Entropy of unique proximal adversarial labels per class, averaged across a balanced sampling of correctly labeled test images.
Discussion and related work
Adversarial subspaces: Previous research has extensively explored subspaces near adversarial perturbations [e.g. 31, 11, 32, 13, 12, 33]. Fawzi et al. [2016] studied the nearest error point in random subspaces, while others analyzed classifier behavior through random perturbations near adversarial examples [10, 12]. Gilmer et al. [2019] and Hendrycks and Dietterich [2019] examined worst-case and general out-of-distribution (OOD) errors, highlighting adversarial inputs as examples of the former.
While these works suggest that local neighborhoods around adversarial examples are often adversarial, indicating a high-rank error space, they leave unanswered questions about the multi-dimensional decision boundary near natural inputs. Our method addresses this gap by identifying subspaces where proximal decision boundary geometry can be measured, leading to key insights:
Proximity assessment: The decision boundary is close to natural inputs within a large subspace, with distances increasing smoothly and slowly as dimensionality grows. This contradicts hypotheses that adversarial examples are isolated [8] or caused by sharp boundary protrusions [35].
Boundary curvature: Adversarial training linearizes boundaries proximal to training images more than distal boundaries, while naturally trained network boundaries tend to exhibit negative curvature when proximal to natural inputs.
Adversarial class composition: Adversarial training produces a more uniform class distribution, indicating greater utilization of high-dimensional space but less alignment with semantic proximity.
Our findings align with studies on adversarial robustness and input dimensionality [36, 37], showing consistent trends across datasets. They also support evidence for large adversarial subspaces, as suggested by works on combining universal adversarial directions [38, 39]. Prior work has used decision boundary measurements to interpret neural network behavior [40] but did not address multidimensional boundary geometry, which we explore in detail. Further discussion of prior approaches to defining adversarial subspaces can be found in Appendix § 13.
Neuron response curvature: Analysis of response curvature has a rich history in neuroscience [see 41 for a review] and computational models of neural activity [e.g. 42, 43, 44, 45]. In the context of ANNs, Golden et al. [ [46] employed principal curvature decomposition to study neuron selectivity and invariance, while Paiton et al. [ [47] connected neuron response curvature to adversarial robustness. Our work differs by focusing on decision boundary curvature rather than individual neuron response curvature.
To the best of our knowledge, our work constitutes the first comprehensive measurement of the decision boundary curvature for nonlinear neural networks. We improve on previous work [31, 48, 21, 49, 46, 40, 50, 51] by providing complete details of the derived formulas in § 2.2 and demonstrating that our solution is uniquely correct by measuring the level-set curvature of simple Riemannian manifolds in Appendix § 14. We also corroborate and extend the findings of Moosavi-Dezfooli et al. [ [21], demonstrating a positive correlation between adversarial vulnerability and decision boundary curvature, with additional comparisons across adversarial subspaces, random subspaces, and the full input space.
Conclusion
We introduce a novel method for measuring the geometry of neural network decision boundaries, presenting the first evidence that decision boundaries are close to natural inputs within large subspaces across architectures and datasets (Figure 2). This supports the hypothesis that adversarial examples occupy dense regions of input space [17, 11, 18, 12] while refuting alternative explanations [8, 35]. Our method improves the accuracy of 2D visualizations (Figure 3), defining subspaces (Appendix § 13), and boundary curvature estimation (Appendix § 14). We provide new evidence that adversarial training linearizes decision boundaries (Figure 4) [52, 49, 53, 54, 55, 56], along with information on its impact on class label composition near natural inputs (Figure 5).
Applications of our work include semi-supervised relabeling [57], architecture design [58], and robustness enhancement [51] (Tian et al. 2021). Furthermore, our framework supports quantitative, falsifiable interpretability, essential for deploying neural networks in high-stakes applications [60, 61, 62, 63]. Future directions include large-scale analysis across architectures, particularly with non-smooth activations [e.g. 65] (ReLU or stochastic, Li et al. 2019b), and investigating relationships between adversarial directions, boundary curvature, overfitting [66], and adversarial feature perceptual quality [67].
Author Contributions
All figures and the first draft were written by D.P. and D.S., although all authors contributed to revisions. The subspace finding algorithm was developed by D.S., and the optimization procedure was developed by D.S. and M.K. All curvature related experiments were conceived by D.P., and written by D.P. and M.K. All other experiments were written and conducted by D.S. The theoretical analysis of boundary curvature was conducted by Z.C., D.P., and M.K. The idea to investigate subspaces where the decision boundary is closest was first proposed by M.B. The project was led by M.B.
Acknowledgments
We would like to thank James Golden for early guidance, Sven Gowal for providing extra trained networks, Wieland Brendel and Yash Sharma for valuable feedback, as well as Jonas Rauber for his contributions on early iterations of the project. Our work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC number 2064/1 – Project number 390727645; and The German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B.
Full derivation for hypersurface curvature
We use definitions that are commonly found, and our notation follows Lee [2013, 2018] for differential geometry, and Magnus and Neudecker [1999] for linear algebra.
Definition of terms
In order to derive our main curvature expressions for a graph submanifold, we first introduce some notation and recall some concepts.
Tangent Spaces If is an -dimensional smooth manifold then the tangent space at is a vector space of derivations at , which is isometric to . In particular, is the set of directional derivative operators acting on the smooth real functions .
The disjoint union of all of the tangent spaces is the tangent bundle . If we equip with a smooth, bilinear, positive definite, symmetric form then is said to be a Riemannian manifold and the Riemannian metric tensor.
When it is unambiguous, denote by the smooth mappings (for some topological space ). Most often we will refer to , and represent elements of with the letters .
Differentials If and are manifolds and is differentiable, then the differential of at is the linear mapping denoted , where the action on tangent vectors is defined by
That is, the rate of change of the function in the direction at the point . As is common practice, we often omit the subscripts for clarity. The linear operator is a linear functional when in the above definition is , and at each there exists a vector . So that
and is the associated vector field.
Connections A connection is an operator , linear in its first argument, and satisfying a product rule in its second. This defines a connection in each tensor bundle,4 which we overload with [69 Prop. 4.15]. Importantly for us, we have for all (for any connection).5 A Riemannian manifold always possesses a Levi–Cevita connection [69 §5], denoted by , which is the unique connection satisfying
To we associate the Hessian operator , which is a symmetric -tensor field [69 Ex. 4.22] with for . We have [via Lee, 2018, Prop. 4.21]
Using the compatibility of with the metric we arrive at the following expression for the Hessian, which will be useful later
Submanifolds
If , are two Riemannian manifolds with , then an injection is called an inclusion mapping. If the inclusion is a smooth isometry, that is
equivalently, , then is called a Riemannian submanifold of .
Using the inclusion map differential, we embed in . If , are of dimensions , then occupies an dimensional subspace of for each . The normal bundle is formed from the orthogonal complement of at each , and is defined in a similar way to the tangent bundle, denoted and having a dimension of . For vectors we have the decomposition with and .
Some care must be taken when taking covariant derivatives in with the ambient connection, formally
where the overlines denote arbitrary smooth extensions of and to open neighbourhoods of in .
Second Fundamental Form Let be an embedded submanifold. The second fundamental form is the (bilinear) mapping with . If we equip with a normal field , then the scalar second fundamental form is
For the remainder we assume is fixed. The shape operator is the endomorphism obtained from that satisfies
that is, by raising an index.
Curvature Let be the shape operator associated to a normal field on an embedded submanifold of dimension . Then at , the (real) eigenvalues of are called principal curvatures and the eigenvectors are called principal directions at . The Gaussian curvature is the product of the principal curvatures, , and the mean curvature is their average, . Assume and are Riemannian manifolds with an inclusion mapping and has the submanifold structure. Suppose possesses a shape operator (or any other linear endomorphism) . The submanifold structure then gives us an operator , the pullback of by , via
Remark 1 (Remark: Euclidean subspace.). We consider the special case of the Riemannian manifold . Suppose is orthogonal () and let be a linear operator. Then is an inclusion map and the subspace pullback of to is
We refer to the curvature quantities and directions of with the subspace prefix. For example, subspace principal curvature.
Hypersurfaces
Let be an embedded submanifold of . Suppose is a local defining function for , that is, for an open . Choose arbitrary . From the connection product rule [69 p. 89]
Whence
The zero in the underbrace follows because and . The Weingarten equation [69 Thm. 8.13(c] yields , giving us
The Curvature of a Graph Manifold
The notation and conventions of this section closely follow [69 §8]. Let be the graph of a smooth function , that is, . Since is smooth is indeed a manifold [68 Ex. 1.30, p. 20]. We regard as a Riemannian submanifold of , with the inclusion map .
The function is a global defining function for with . Denote by the vector of partial derivatives of , and by and , the matrices of second order partial derivatives of and , all with respect to usual coordinate frames, , on and . Then
At the differential has the Jacobian matrix
We identify with its Jacobian and with their parameterizations under the coordinate frame, so that we may use the notation of matrix multiplication.
In dropping the -subscripts, the left inverse, , is has the parameterization
Thus
whence
We can now compute the shape operator of associated to the unit normal field . Pick arbitrary . From (7) there is
and, using the bilinearity of of we have
This, together with (8) yields the shape operator:
The Curvature of a Level-Set Manifold
We now consider the case where is the level-set of a smooth function so that for some we have . The following theorem can be used to ensure the level-sets of do indeed posses a manifold structure, which is a simplification of the constant rank level-set theorem Lee, [68] for our setting.
Theorem 1 (Theorem.). Let be a smooth function having constant rank . Then the level-sets of are all smooth manifolds of codimension .
We use the same construction as Lee [ [68] to calculate the Gaussian and mean curvature at a point .
From the implicit function theorem Lee, [68] there are open connected neighborhoods with and a smooth function so that is the graph of . That is, .
Consequentially the gradient of has the structure
Let and . Then for and
and
where every function on the right hand side is evaluated at either the point or (depending on the function).6 If we let , and be, similarly, the matrix of cross-partial derivatives, there are the more compact expressions for (12) and (13):
and
where the sum is to be interpreted as the matrix of pairwise sums of the elements of : . Then, as in Section A.0.4, is a local defining function for on . The local graph parameterization provided by is with , and its differential has the parameterization
Finally, mutatis mutandis, the shape operator is given by (10).
Remark 2 (Numerical considerations). In order to compute the level-set curvature we needed to apply the implicit function theorem. However, it may be difficult numerically to stabilize the inverse operation in (11). To remedy this problem, one may apply a change of basis before computing the quantities (12) and (13). Let be an orthogonal change of basis operator. Then we apply all our operations to .
Differentiating yields the revised quantities
Summary
In order to study the curvature over a given subspace we introduced the subspace pullback of
where is an orthogonal projection, so that the image of spans the subspace of in question. This definition of essentially restricts the shape operator to a linear subspace of the tangential space and is motivated by the principles of differential geometry. The pullback shape operator allows us to compare the amount of curvature in the subspace to that of the entire response space.
Dataset and models
For the adversarial subspace analysis we used two identical ResNet-50 architectures [22, 71]7 and two identical WideResNet-70 architectures [19]8. For each architecture, one model was trained on the unmodified CIFAR-10 dataset [23]. For the second model we additionally incorporated adversarial examples in the training process, widely known as adversarial training. A training perturbation size of 0.5 was used. To perform the curvature analysis, we used twice-differentiable ELU activation functions [72] for both models, which had little impact on the clean or adversarial accuracies.
In the analysis we include 100 CIFAR-10 test images, class-balanced, all unseen during training, and correctly classified by both models. The optimization algorithm stopped after finding 50 orthogonal adversarial vectors for all 100 test images. For the curvature analysis we used a subset of 50 class-balanced images and 10 adversarial directions per image.
Additionally, we only considered one seed per CIFAR-10 model because we found that the analysis on the MNIST trained models had little variation across seeds (see Appendix § 12). The accuracy of both models at different perturbation lengths and dimensions is shown in Table 2. Here, the first line of each row corresponds to the accuracy of the naturally trained model, whereas the second line is referring to the adversarially trained model.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
| 0 | 95.25% / 90.83% | 95.25% / 90.83% | 95.25% / 90.83% | 95.25% / 90.83% | 95.25% / 90.83% | 95.25% / 90.83% | 95.25% / 90.83% | 95.25% / 90.83% | 95.25% / 90.83% | 95.25% / 90.83% |
| 0.1 | 51.43% / 86.29% | 62.86% / 90.83% | 70.48% / 90.83% | 76.2% / 90.83% | 79.06% / 90.83% | 80.01% / 90.83% | 81.92% / 90.83% | 81.92% / 90.83% | 82.87% / 90.83% | 83.82% / 90.83% |
| 0.25 | 1.91% / 79.93% | 4.76% / 89.01% | 5.72% / 89.92% | 6.67% / 90.83% | 7.62% / 90.83% | 8.57% / 90.83% | 9.52% / 90.83% | 9.52% / 90.83% | 9.52% / 90.83% | 10.48% / 90.83% |
| 0.5 | 0.0% / 65.4% | 0.0% / 84.47% | 0.0% / 86.29% | 0.0% / 87.2% | 0.0% / 87.2% | 0.0% / 87.2% | 0.0% / 87.2% | 0.0% / 87.2% | 0.0% / 87.2% | 0.0% / 87.2% |
| 1 | 0.0% / 36.33% | 0.0% / 54.5% | 0.0% / 59.04% | 0.0% / 61.76% | 0.0% / 65.4% | 0.0% / 69.03% | 0.0% / 69.94% | 0.0% / 70.85% | 0.0% / 72.66% | 0.0% / 73.57% |
| 2 | 0.0% / 2.72% | 0.0% / 5.45% | 0.0% / 10.9% | 0.0% / 12.72% | 0.0% / 13.62% | 0.0% / 14.53% | 0.0% / 15.44% | 0.0% / 15.44% | 0.0% / 16.35% | 0.0% / 17.26% |
Table 2. Accuracies of CIFAR models at different perturbation lengths
We repeated many of the experiments on the MNIST dataset [73] and found largely consistent results (see Appendix § 11).
Additional CIFAR-10 results
Here we show additional analysis to complement our findings on the CIFAR-10 dataset.
Figure 6 shows five adversarially perturbed images and their corresponding original image found for the two ResNet-50 models. We can see that the perturbation length is increasing with dimension number. Interestingly, while the adversarial categories may not be semantically adjacent to the clean classes, the adversarially trained model seems to require more semantically meaningful perturbations, e.g. the images of boats in the top row, misclassified as cars, are perturbed to have elements looking like wheels. We can also see that there is no clear, consistent -length of an adversarial attack that makes an image perturbation ambiguous for a human observer. For example, the robust model example in the first (ship) row of the adversarial 11 column pair has a perturbation length of 1.12 and could be mistaken as a car on a first glance. In contrast, the robust model example in the fourth (horse) row of the
adversarial 1 column pair is still clearly identifiable as a horse with a larger perturbation length of 1.2. This illustrates that the direction of the perturbation is as important as the perturbation magnitude itself, as was previously reported by [17, 67], although assessing the “perceptual distance” is an open area of research.
As was described earlier, we uniformly randomly sampled directions within adversarial subspaces to measure the distance to the decision boundary. Figure 7 shows how many of the samples pointed to decision boundaries that were outside of the allowed input pixel bounds.
Figure 8 shows that perturbation lengths are increasing sublinearly and monotonically with increasing number of adversarial dimensions. For the naturally trained model there is also a much larger increase of adversarial dimension with increasing distance to distance to decision boundary. This figure is different from e.g. Figure 2 in that it displays the lengths of the adversarial subspace axes, not the lengths of the random perturbations within these subspaces. Note that in the right panel of this figure both models plateau at 50 dimensions because we stopped the optimizer at this point.
Figure 8. Perturbation lengths and dimensionality. (left) The subfigure shows the distribution of the n-th found adversarial perturbations across images. (right) We tested at distance increments of 0.001 how many adversarial dimensions were found at that distance per image. The lines represent the means over all test images and the color shaded areas show the 10th and 90th percentiles.
We also provide additional, and more detailed information on the distributions of distances to the decision boundary of randomly sampled vectors in adversarial subspaces in Figure 9. Each subplot shows a subspace of increasing dimensionality, ascending left-to-right, then top-to-bottom. We observe that the variance of the distances to the decision boundary of randomly sampled vectors is smoothly increasing with dimensionality.
Figure 9. Distributions of distances to decision boundary. (A,B) Distributions for natural model. (C,D) Distributions for adversarially trained model. (A-C) We show the complete set histograms corresponding to Figure 2 dimensions. (C,D) Additionally, we provide the mean-normalized histograms to corresponding to the histograms in A and C to emphasize the smooth increase in variance of distances with increasing dimensionality.
Additional model comparison
To show that our findings hold for different models, we present results for two different architectures: a ResNet-50 [22] and a WideResNet-70-16, of which the latter is the state-of-the-art adversarially trained network for CIFAR-10 images [19]. Both architectures were trained with and without adversarial perturbations, resulting in 4 models overall. Figure 10 through 14 repeat analysis to compare natural and adversarial trained models, but instead compare models of different architectures and constant training procedure.
Figure 10. Distances to the decision boundary. Compare with Figure 2.
Figures 10, 11, and 14b show a comparison of the two naturally trained models. These figures indicate that there is little difference between the two model architectures regarding distance to the decision boundary and perturbation length over dimensions. Figures 12, 13, and 14a show a comparison of the two adversarially trained models. Here, the figures show that perturbation lengths as well as distances to the decision boundary are larger for the WideResNet-70-16 compared to the ResNet-50. This is expected, as the WideResNet has higher adversarial accuracy and thus larger perturbations are needed to fool it. At the level of these aggregate statistics, the hyperparameter differences in adversarial training (Figure 12) have more of an effect than the model architecture differences themselves (Figure 10), although all differences are much less pronounced than the differences between adversarially and naturally trained networks (Figure 2).
Figure 12. Distances to the decision boundary. Compare with Figure 2.
The box plots depicting the distances to the decision boundary in Figure 10 and Figure 12 show that, for both architectures, the adversarial subspace is large, continuous, and increases sublinearly in volume as the dimensionality is increased. The observation that this finding is agnostic to model architecture is further supported by the similarity in distributions of samples measuring the distance to decision boundary that can be seen on the right hand side of the same figures. For Figure 15 we repeated the curvature analysis on the natural and robust neural networks, although we limited the experiment to 10 images and 8 dimensional adversarial subspaces due to the increase in computational cost associated with the larger model size. Comparing this figure against Figure 4 shows that the mean and principal curvature profiles are largely consistent. Finally, Table 3 and Figure 16 demonstrate that adversarial training results in an additional diversity of nearby classes within adversarial subspaces for the WRN-70 network, although the effect is less pronounced than for the RN-50 network. Thus, the observed trends and conclusions drawn from the analysis provided in this study hold across architectures.
| Label | Natural | Robust |
| airplane | 1.83 | 1.84 |
| car | 1.47 | 1.44 |
| bird | 1.69 | 1.63 |
| cat | 1.53 | 1.41 |
| deer | 1.55 | 1.80 |
| dog | 1.49 | 1.61 |
| frog | 1.23 | 1.38 |
| horse | 1.65 | 1.95 |
| ship | 1.78 | 1.77 |
| truck | 1.68 | 1.79 |
Table 3. Entropy of unique proximal adversarial labels per class, averaged across a balanced sampling of correctly labeled test images. For 6 out of 10 categories the entropy is increased with adversarial training. Results are for the WRN-70 network. Compare with Table 1.

Figure 15. Decision boundary curvature. WRN-70 curvature analysis. Compare with Figure 4.
Figure 16. Adversarial Class Composition. WRN-70 adversarial class analysis. Compare with Figure 5.
Additional MNIST results
We conducted the same experiments described in § 3 on networks trained with the MNIST dataset. As a test set we used 500 images (50 per class), unseen in training and correctly labeled by all investigated models.
In Figure 20 we can see that perturbation lengths are much larger for MNIST than for CIFAR-10, although the relative increase in distances for the robust network is still present. The optimization was in general more difficult with MNIST, likely due to the fact that the images all live very close to the allowable input bounds. Specifically, the algorithm found less than 30 adversarial dimensions for more than 95% of the test images (Figure 24), to be compared with finding >50 dimensions for 100% of the CIFAR-10 test images.
Figure 20. MNIST perturbation lengths and dimensions. Compare with Figure 8.
Figure 24. Optimization consistency on MNIST. The bars show number of samples for which at least dimensions were found during optimization. Different colors indicate distinct seeds for model training.
That it is harder to fool the MNIST model can also be seen in Table 4: the accuracies of both models are much higher for the same perturbation lengths than the accuracies of the CIFAR-10 models. For the curvature analysis, on the other hand, we found that the overall values were larger on CIFAR-10. Additionally, the adversarially trained model had more decision boundary curvature within adversarial subspaces on MNIST than on CIFAR-10.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
| 0 | 98.59% / 98.56% | 98.59% / 98.56% | 98.59% / 98.56% | 98.59% / 98.56% | 98.59% / 98.56% | 98.59% / 98.56% | 98.59% / 98.56% | 98.59% / 98.56% | 98.59% / 98.56% | 98.59% / 98.56% |
| 0.5 | 90.51% / 97.77% | 97.41% / 98.56% | 98.0% / 98.56% | 98.39% / 98.56% | 98.39% / 98.56% | 98.39% / 98.56% | 98.37% / 98.56% | 98.35% / 98.56% | 98.29% / 98.56% | 98.23% / 98.56% |
| 1 | 60.53% / 94.03% | 90.31% / 98.36% | 95.83% / 98.56% | 97.01% / 98.56% | 97.4% / 98.56% | 97.58% / 98.56% | 97.94% / 98.56% | 97.86% / 98.56% | 97.69% / 98.56% | 97.51% / 98.56% |
| 2 | 1.77% / 68.99% | 18.34% / 89.3% | 47.91% / 95.01% | 71.58% / 96.59% | 83.54% / 97.57% | 88.51% / 97.69% | 91.63% / 97.73% | 93.0% / 97.72% | 93.8% / 98.56% | 93.55% / 98.56% |
| 4 | 0.0% / 0.59% | 0.0% / 9.46% | 0.2% / 27.99% | 4.14% / 46.52% | 15.05% / 65.77% | 26.41% / 76.32% | 32.43% / 82.59% | 38.61% / 83.94% | 37.46% / 85.42% | 32.74% / 80.16% |
| 6 | 0.0% / 0.0% | 0.0% / 0.0% | 0.0% / 0.2% | 0.39% / 7.69% | 1.98% / 24.69% | 7.86% / 37.51% | 10.45% / 47.63% | 18.94% / 48.86% | 16.48% / 54.54% | 16.55% / 40.74% |
Table 4. Accuracies of MNIST models (seed 0) at different perturbation lengths
We noticed that most of the MNIST adversarials lie in a thin regions between the decision boundary and the valid pixel box boundary (compare Figure 19 against Figure 3). Figure 17 shows that this leads to large rates of out of bounds samples when measuring the distance to the decision boundary. For example, 90% of the time the decision boundary is outside the valid pixel range in the 6 dimensional adversarial subspace of the naturally trained model.
Figure 17. Ratio of out of bounds samples. Compare to Figure 7.
Figure 19. MNIST boundary visualizations. This figure is the MNIST version of Figure 3. Note that for MNIST, the decision boundary lies outside the valid pixel range more often than for CIFAR.
Figure 11. Perturbation lengths and dimensionality. Compare with Figure 8.
Seed consistency
We trained the natural and robust MNIST models with five different seeds and extracted adversarial subspaces for each model to examine the influence of training seeds on our results. Figure 24 shows that there is slight variation in number of adversarial subspace directions found by the optimizer. However, the overall trend of bar heights is similar across seeds and there are no significant outliers.
To further validate our results, we repeated the experiment for Figure 18 (measuring the length of the adversarial vector as one increases dimensionality on the MNIST trained networks) across five different seeds per training type. The results are shown in Figure 25. The variation across seeds for a single image is smaller than the variation across images for a single seed, which can be seen by comparing Figure 25 to Figure 18. In general, we also found that the results and trends were consistent between MNIST and CIFAR-10 networks (with differences noted in Appendix § 11). Thus, we considered one seed per CIFAR-10 model as sufficiently representative.
Figure 18. MNIST distances to the Decision Boundary. We repeated the distance analysis from Figure 2 on the MNIST dataset Only test images with at least 8 adversarial dimensions were included in this Figure.
Figure 25. Distance consistency on MNIST. Box plots show the length of the adversarial vector as one increases dimensionality. The data is computed for a single image, where the variation comes from five different seeds used to initialize the networks before training. Note that the variation is much smaller than the variation across images for a single seed, as can be seen in Figure 18.
Figure 13. Perturbation lengths and dimensionality. Compare with Figure 8.
We also investigated the influence of random number generation relevant for sampling within adversarial subspaces. As an empirical test, we varied the number of samples and measured the change in mean distance to the decision boundary as one increases the number of samples. This was repeated for 100 different 50-dimensional subspaces (i.e. subspaces for 100 different test images) with the naturally trained CIFAR network. Figure 23 shows that the absolute change decreases steadily to an average distance of 0.004. Thus, the mean variation from 100 samples is much smaller than the general distance scales we are measuring throughout the paper. We also measured the variation in distances across random seeds for the sampling process. To do this, we used 10 different seeds to generate 100 samples in a 50-dimensional subspace corresponding to one test image. For each seed, we compute a mean distance, averaged across 100 samples, giving us 10 values. The mean distance across seeds was 0.0933, and the standard deviation was 0.0007. The standard deviation is less than 1% of the mean and thus sufficiently small to consider an estimate with 100 samples as representative.

Figure 23. Sampling size in adversarial subspaces. The line shows the mean running difference in distance to the decision boundary means across 100 samples as one increases the sample size from 5 to 100. The shaded area depicts the standard deviation from the mean line.
Figure 14. Ratio of out of bounds samples. Compare with Figure 7.
Comparisons to related work: Orthogonality and clipping
One obstacle in finding proper adversarial subspaces spanned by orthogonal vectors is assuring that perturbations do not cause pixel values to be out of the valid input boundaries . We noticed that many studies solve this problem by element wise pixel clipping. Here, we show that pixel clipping is problematic when talking about the dimensionality of subspaces. Element wise pixel clipping is a non-linear operation performed on a vector and can therefore change its direction in the input space and its relative angle to other, fixed vectors.
To show that clipping has a non-negligible effect, we investigated relative angles of basis vectors that span (sub-) spaces for 100 MNIST test images relevant for two studies.
First, in their work, Tramèr et al. [ [13] propose the GAAS method. GAAS estimates the dimensionality of adversarial subspaces by defining an orthogonal set of vectors aligned with an attack vector to an input image. However, before passing the orthogonal basis vectors to the model, a clipping operation is performed. Figure 26 (left) shows that this clipping operation causes the input vectors to be non-orthogonal – orthogonality would constitute an angle of or . In fact, for the 100 sample images we tested, there is not a single vector pair that is completely orthogonal. The author’s conclusions regarding the dimensionality of adversarial subspaces and transferable subspaces are therefore grounded on ill-defined subspaces, spanned by non-orthogonal vectors.
Figure 26. Clipping causes non-orthogonality. The histograms show angles between vectors that define “subspaces” in the GAAS method proposed by Tramèr et al. [2017] (left) and distance to decision boundary estimations by He et al. [2018] (right).
In another study, He et al. [ [12] estimate the distance to the decision boundary for the complete input space of MNIST. For that, a basis of 784 vectors, spanning the whole input space, is randomly generated. Again, the inputs are clipped before they are passed to the model leading to the same problem as described earlier. And as before, there are no orthogonal vector pairs, and angels are accumulated at much smaller values than as can be seen in Figure 26 (right). We therefore note that it is not guaranteed that the input space is
evenly covered and that edges of the 784-dimensional hypercube (in the case of MNIST) are arguably overrepresented.
Comparisons to related work: Decision boundary curvature
| Reference | Derivation | Generating matrix | Dimension | |
| Poole et al., 2016 | None | |||
| Moosavi-Dezfooli et al., 2018 | None | Not given | ||
| Moosavi-Dezfooli et al., 2019 | None | Crossentropy Hessian | - | |
| Moosavi-Dezfooli, 2019, p. 90 | None | |||
| This work | Appendix A | (10) using (13) | - |
Table 5. Decision boundary curvature formulas present in the literature for a decision function . The generating matrix column refers to the matrix used to yield eigenvalues for the principle curvatures, and is roughly comparable to the shape operator.
This section provides a detailed comparison of our analysis against previously published works on the principal curvature analysis applied to decision boundaries. We include table 5 for a quick summary.
Figure 21. MNIST example adversarials. Compare to Figure 6.
Figure 6. CIFAR-10 example adversarials. The panels show adversarial examples for a given input image. Column pairs are sorted by adversarial dimension. Within a pair, each column includes examples for the natural and robust models, respectively. The labels below the images indicate the class assigned by the model. The numbers in brackets are the -norms of the respective images.
The use of differential geometry to understand the non-linear response properties of model neurons has a long history in computational neuroscience, with early contributions made by Zetzsche and Barth [1990]. More recently, several works have specifically used Reimannian geometry to measure the principal curvature of response manifolds [31, 48, 21, 49, 46]. As we discuss in Appendix § 8, certain smoothness assumptions are required for such an analysis. Our present study, as well Poole et al. [2016] and Golden et al. [2019], avoid this problem by using twice-differentiable activation functions. In the case of Golden et al. [2019], this was done without retraining the network, while Poole et al. [2016] used nets with random weights. We instead trained the twice-differentiable networks from scratch. Moosavi-Dezfooli et al. used a finite-difference approximation to avoid the problem [described in Moosavi-Dezfooli et al., 2019, eq 1 and surrounding discussion]. Another high-level difference, discussed in detail in Appendix § 8, is the choice of measuring the response graph curvature (which is dimensional for dimensional inputs) or the level-set curvature (which is dimensional for dimensional inputs) of a given function. This difference was discussed in [46 , at the end of Section 4 and in Figure 4], although their technical derivations are limited only to the graph formulation. The other studies did not explicitly discuss this difference. Importantly, though, the decision boundary is the level-set of the logit difference function, as defined in Equation (1).

Figure 22. MNIST decision boundary curvature. Compare with Figure 4
Several works from Moosavi-Dezfooli and colleagues explore decision boundary curvature and its relationship to adversarial robustness [31, 76, 21, 49] (Moosavi-Dezfooli 2019). Within these works they provide an inspired perspective on explaining adversarial robustness in terms of the curvature of decision boundaries. Almost all of the work is with respect to universal adversarial perturbations, which share a common subspace [18], although Fawzi et al. [ [31] alternatively look at random subspaces. Their universal adversarial perturbation subspaces differ from our adversarial subspaces in that theirs are defined by orthogonal directions that transfer well across inputs instead of those that point to the nearest decision boundary for a given input. Despite these differences, the stated quantity of interest, that is, the principle curvatures at a point on the decision boundary is the same in their work as well as ours.
However, each of these papers have a slightly different analytic formula for estimating the curvature of the boundary within a given subspace [Fawzi et al., 2016, Moosavi-Dezfooli et al., 2017b, or in the high-dimensional input space [21, 49]. We found the most complete description to be given in Moosavi-Dezfooli [2019], which summarizes a majority of the earlier work and applies the decision boundary curvature analysis to both subspaces and the high-dimensional input space. Therein, Moosavi-Dezfooli [ [74] state the following formula, without derivation, for the shape operator (in the notation of Appendix § 8):
where is the Hessian of the boundary function, , and is described as a projection operator. This formulation is similar to that used by [48], with the only difference being that they use normalized gradients in the projection operator.
Our formulation differs from the above in several ways, the most important of which is that they claim to transform the Hessian by projecting it onto the tangent space (as written above), while we apply the implicit function theorem to parameterize the decision boundary as the graph of a function (see Appendix Section A.0.5, Equation (11)). For computing the curvature of the decision boundary, exclusively projecting the Hessian is not going to result in the correct calculations, since for the decision boundary (as defined in § 2.2, Equation (1)) you need a different normal than for the graph of the response function. More specifically, the normal used for computing the shape operator of the graph surface is different from that for computing the shape operator of the decision boundary. The latter has to be exclusively in the pixel space, while the former always has a component in the direction of the function response. Projecting the Hessian can restrict the computed curvatures in a certain subspace of directions, but it cannot change the computed curvatures in said subspace. Another important difference is their projection operator is computed directly from the gradient, and thus has a dimensionality equal to the input dimensionality. To calculate the curvature of a decision boundary, one needs a level-set formulation, which should have one fewer dimensions than the input space.
The consequence of this difference can be made clear when looking at the curvature of an -dimensional sphere, as defined by the quadratic function:
where is the input point vector and is the sphere’s Hessian (i.e. an identity matrix in ). Given this function, the principal curvatures of the level-set at some radius, , are all known to be (up to a sign change) . However, we show in Table 6 that the formulations provided by Moosavi-Dezfooli [2019] and Poole et al. [2016] produced curvatures that are often several orders of magnitude off from the correct values the case of high-dimensional inputs. It is possible that these previous works compute the correct curvature for their specific situation studied, although this would be difficult to verify without a full derivation of the curvature quantities and our sphere test is closely matched to the decision boundary level-set curvature that we measure. Due to the dimensionality issue explained above, we computed the average error across all principal curvatures, regardless of the number of curvatures reported, as they should all equal .
| dimensionality-radius | Moosavi et al. | Poole et al. | Ours |
| 02 - 00.1 | 7.6e-01 | 5.0e+00 | 0.0e+00 |
| 02 - 01.0 | 2.4e+01 | 5.0e-01 | 0.0e+00 |
| 02 - 10.0 | 3.1e+04 | 5.0e-02 | 1.3e-17 |
| 05 - 00.1 | 7.2e-01 | 2.0e+00 | 2.2e-15 |
| 05 - 01.0 | 7.2e+01 | 2.0e-01 | 2.2e-16 |
| 05 - 10.0 | 7.9e+04 | 2.0e-02 | 3.8e-17 |
| 10 - 00.1 | 6.4e-01 | 1.0e+00 | 2.3e-15 |
| 10 - 01.0 | 1.5e+02 | 1.0e-01 | 2.2e-16 |
| 10 - 10.0 | 1.5e+05 | 1.0e-02 | 4.4e-17 |
Table 6. Calculations from alternative attempts find incorrect principal curvature for a level-set on a sphere of varying dimensionality and radius. The quantity reported is the mean of the absolute difference between each computed principal curvature and the inverse radius.
We additionally tested a graph formulation (defined in Section A.0.4), using a one-sheeted hyperboloid. This formulation lives in the full space, so there is a principal curvature per input dimension. However, we show similar errors in Table 7. Note, that the formulation from Golden et al., [46] matched our own, although they did not provide an equation for mapping this to the level-set curvature. Our own implementations were correct with high precision on these and other experiments with known functions, all of which can be replicated from our provided code. Finally, while the subspace projection methods proposed by Fawzi et al., [31] and Moosavi-Dezfooli et al., [76] potentially provide metrics related to curvature in the subspaces they defined, we chose to instead derive our pullback method described in Equation (14) from primary sources, as explained in Appendix § 8.
| Analysis type | Absolute error |
| Moosavi et al. | 6.8e+00 |
| Poole et al. | 1.4e-03 |
| Golden et al. | 4.1e-10 |
| Ours | 4.1e-10 |
Table 7. Calculations from alternative attempts find incorrect Gaussian curvature (i.e. the product of the principal curvatures) for a Hyperboloid graph.
References
- [1]Fuxun Yu, Chenchen Liu, Yanzhi Wang, Liang Zhao, and Xiang Chen. Interpreting adversarial robustness: A view from decision surface in input space. arXiv preprint arXiv:1810.00144, 2018.arxiv.org/abs/1810.00144
- [2]Francisco Rodrigues, Mateus Espadoto, Roberto Hirata, and Alexandru C Telea. Constructing and visualizing high-quality classifier decision boundary maps. Information, 10(9):280, 2019.DOI
- [3]Raphael Féraud and Fabrice Clérot. A methodology to explain neural network classification. Neural networks, 15(2):237–246, 2002.pubmed.ncbi.nlm.nih.gov/12022511
- [5]Polina Golland. Discriminative direction for kernel classifiers. In Advances in neural information processing systems, NeurIPS, volume 14, pages 745–752, 2001.DOI
- [7]Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pages 387–402. Springer, 2013.
- [8]Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR, 2014. URL https://openreview.net/forum?id=klr_MTHMRqGj.arxiv.org/abs/1312.6199
- [9]Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are a natural consequence of test error in noise. In International Conference on Machine Learning, ICML, pages 2280–2289. PMLR, 2019.
- [10]David Warde-Farley and Ian J Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow, editors, Perturbations, Optimization, and Statistics, page 311. MIT Press, 2016.
- [11]Pedro Tabacof and Eduardo Valle. Exploring the space of adversarial images. In 2016 International Joint Conference on Neural Networks, IJCNN, pages 426–433. IEEE, 2016.arxiv.org/abs/1510.05328
- [12]Warren He, Bo Li, and Dawn Song. Decision boundary analysis of adversarial examples. In International Conference on Learning Representations, ICLR, 2018.
- [13]Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv:1704.03453, 2017.arxiv.org/abs/1704.03453
- [14]Grzegorz Swirszcz, Brendan O’Donoghue, and Pushmeet Kohli. Visualizations of decision regions in the presence of adversarial examples. In ICLR Debugging Machine Learning Models Workshop, 2019.
- [15]Hamid Karimi and Tyler Derr. Decision boundaries of deep neural networks. In 2022 21st IEEE international conference on machine learning and applications (ICMLA), pages 1085–1092. IEEE, 2022.DOI
- [16]Ahmed Imtiaz Humayun, Randall Balestriero, Guha Balakrishnan, and Richard G Baraniuk. Splinecam: Exact visualization and characterization of deep network geometry and decision boundaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3789–3798, 2023.arxiv.org/abs/2302.12828
- [17]Ian J Goodfellow, Jonathan Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.DOI
- [18]Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, CVPR, pages 1765–1773, 2017a.
- [19]Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.arxiv.org/abs/2010.03593
- [20]Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4312–4321. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ijcai.2021/591. URL https://doi.org/10.24963/ijcai.2021/591. Survey Track.
- [21]Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, Pascal Frossard, and Stefano Soatto. Robustness of classifiers to universal perturbations: A geometric perspective. In International Conference on Learning Representations, ICLR, 2018.
- [22]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, CVPR, pages 770–778, 2016.arxiv.org/abs/1512.03385
- [23]Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://www.cs.toronto.edu/~kriz/cifar.html.
- [24]Andreas Wächter and Lorenz T Biegler. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical programming, 106(1):25–57, 2006.DOI
- [25]Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy, pages 39–57. IEEE, 2017.arxiv.org/abs/1608.04644
- [26]Chris Finlay, Aram-Alexandre Pooladian, and Adam Oberman. The logbarrier adversarial attack: making effective use of decision boundary information. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4862–4870, 2019.arxiv.org/abs/1903.10396
- [27]Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Ead: elastic-net attacks to deep neural networks via adversarial examples. In Thirty-second AAAI conference on artificial intelligence, 2018.arxiv.org/abs/1709.04114
- [28]Mervin E. Muller. A note on a method for generating points uniformly on n-dimensional spheres. Commun. ACM, 2(4):19–20, April 1959. ISSN 0001-0782. doi: 10.1145/377939.377946. URL https://doi.org/10.1145/377939.377946.
- [29]Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, ICLR, 2018.arxiv.org/abs/1706.06083
- [30]Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In International Conference on Learning Representations, ICLR, 2020.
- [31]Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Robustness of classifiers: from adversarial to random noise. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, NeurIPS, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper/2016/file/7ce3284b743aefde80ffd9aec500e085-Paper.pdf.arxiv.org/abs/1608.08967
- [32]Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, ICLR, 2017.arxiv.org/abs/1611.02770
- [33]Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations, 2018.arxiv.org/abs/1801.02613
- [35]Tao Yu, Shengyuan Hu, Chuan Guo, Wei-Lun Chao, and Kilian Q Weinberger. A new defense against adversarial images: Turning a weakness into a strength. In Advances in Neural Information Processing Systems, NeurIPS, volume 32, 2019.arxiv.org/abs/1910.07629
- [36]Laurent Amsaleg, James Bailey, Dominique Barbe, Sarah Erfani, Michael E Houle, Vinh Nguyen, and Miloš Radovanović. The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality. In 2017 IEEE Workshop on Information Forensics and Security (WIFS), pages 1–6. IEEE, 2017.DOI
- [37]Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez-Paz. First-order adversarial vulnerability of neural networks and input dimension. In International Conference on Machine Learning, pages 5809–5817. PMLR, 2019.
- [38]Yuan Liu and Stephane Canu. The data-driven transferable adversarial space. In The 16th Asian Conference on Machine Learning (Conference Track), 2024.
- [39]Jordan Patracone, Lucas Anquetil, Yuan Liu, Gilles Gasso, and Stéphane Canu. Linear modeling of the adversarial noise space. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 301–317. Springer, 2024.DOI
- [40]David Mickisch, Felix Assion, Florens Greßner, Wiebke Günther, and Mariele Motta. Understanding the decision boundary of deep neural networks: An empirical study. arXiv preprint arXiv:2002.01810, 2020.arxiv.org/abs/2002.01810
- [41]Tim Gollisch and Andreas VM Herz. The iso-response method: measuring neuronal stimulus integration with closed-loop experiments. Frontiers in neural circuits, 6:104, 2012.DOI
- [42]Christoph Zetzsche and Gerhard Krieger. Nonlinear mechanisms and higher-order statistics in biological vision and electronic image processing: review and perspectives. Journal of Electronic Imaging, 10(1):56–100, 2001.DOI
- [43]Nicole C Rust, Odelia Schwartz, J Anthony Movshon, and Eero P Simoncelli. Spatiotemporal elements of macaque v1 receptive fields. Neuron, 46(6):945–956, 2005.pubmed.ncbi.nlm.nih.gov/15953422
- [44]James R Golden, Kedarnath P Vilankar, Michael C K Wu, and David J Field. Conjectures regarding the nonlinear geometry of visual neurons. Vision research, 120:74–92, 2016.DOI
- [45]Uri Cohen, SueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):1–13, 2020.DOI
- [46]James R Golden, Kedarnath P Vilankar, and David J Field. Selective and invariant features of neural response surfaces measured with principal curvature. bioRxiv, 2019. URL https://doi.org/10.1101/2019.12.26.888933.
- [47]Dylan M Paiton, Charles G Frye, Sheng Y Lundquist, Joel D Bowen, Ryan Zarcone, and Bruno A Olshausen. Selectivity and robustness of sparse coding networks. Journal of Vision, 20(12):10–10, 2020.DOI
- [48]Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29:3360–3368, 2016.arxiv.org/abs/1606.05340
- [49]Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pages 9078–9086, 2019.
- [50]Shiye Lei, Fengxiang He, Yancheng Yuan, and Dacheng Tao. Understanding deep learning via decision boundary. IEEE Transactions on Neural Networks and Learning Systems, 2023.DOI
- [51]Ali Sekmen and Bahadir Bilgin. Manifold-based approach for neural network robustness analysis. Communications Engineering, 3(1):118, 2024. doi: 10.1038/s44172-024-00263-8. URL https://doi.org/10.1038/s44172-024-00263-8.
- [52]Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and Jörn-Henrik Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems, NeurIPS, 32:15390–15402, 2019a.arxiv.org/abs/1911.00937
- [53]Dávid Terjék. Adversarial lipschitz regularization. In International Conference on Learning Representations, ICLR, 2020.arxiv.org/abs/1907.05681
- [54]Anindya Sarkar and Raghu Iyengar. Enforcing linearity in dnn succours robustness and adversarial image generation. In Igor Farkaš, Paolo Masulli, and Stefan Wermter, editors, Artificial Neural Networks and Machine Learning – ICANN 2020, pages 52–64, Cham, 2020. Springer International Publishing. ISBN 978-3-030-61609-0.arxiv.org/abs/1910.08108
- [55]Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472–7482. PMLR, 2019.arxiv.org/abs/1901.08573
- [56]Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33, 2020.arxiv.org/abs/2004.05884
- [57]Barbara Caroline Benato, Alexandru Cristian Telea, and Alexandre Xavier Falcão. Semi-supervised learning with interactive label propagation guided by feature space projections. In 2018 31st Conference on Graphics, Patterns and Images, SIBGRAPI, pages 392–399. IEEE, 2018.DOI
- [58]Paulo E Rauber, Alexandre X Falcao, and Alexandru C Telea. Projections as visual aids for classification system design. Information Visualization, 17(4):282–305, 2018.DOI
- [60]Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.arxiv.org/abs/1602.04938
- [61]Georgios Vlassopoulos, Tim van Erven, Henry Brighton, and Vlado Menkovski. Explaining predictions by approximating the local decision boundary. arXiv preprint arXiv:2006.07985, 2020.arxiv.org/abs/2006.07985
- [62]Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.
- [63]Matthew L Levitt and Ari Morcos. Towards falsifiable interpretability research. arXiv preprint arXiv:2010.12016, 2020.
- [65]Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D Cox, and James J DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. BioRxiv, 2020.DOI
- [66]Thomas Tanay and Lewis Griffin. A boundary tilting perspective on the phenomenon of adversarial examples. arXiv preprint arXiv:1608.07690, 2016.arxiv.org/abs/1608.07690
- [67]Andrew Ilyas, Shibani Santurkar, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in neural information processing systems, NeurIPS, volume 32, 2019.arxiv.org/abs/1905.02175
- [68]John M. Lee. Introduction to Smooth Manifolds. Number 218 in Graduate Texts in Mathematics. Springer, 2 edition, 2013. ISBN 978-1-4419-9981-8 978-1-4419-9982-5.
- [69]John M. Lee. Introduction to Riemannian Manifolds. Springer International Publishing, 2 edition, 2018. ISBN 978-3-319-91754-2.DOI
- [71]Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL https://github.com/MadryLab/robustness.
- [72]Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.arxiv.org/abs/1511.07289
- [73]Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- [74]Seyed-Mohsen Moosavi-Dezfooli. Geometry of adversarial robustness of deep networks: methods and applications. PhD thesis, EPFL, 2019. URL https://infoscience.epfl.ch/record/271933.
- [76]Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, Pascal Frossard, and Stefano Soatto. Analysis of universal adversarial perturbations. ArXiv e-prints, pages arXiv–1705, 2017b.





















