betaFeedback

WorksComputer Science

Robust Deep Prototype Classifiers via Lipschitz Training

Abstract

Robustness and interpretability are interrelated and desirable properties of machine learning models. A model that changes its prediction under imperceptible perturbations behaves inconsistently with human-recognizable features and, hence, is not interpretable. Prototype-based classifiers such as Generalized Learning Vector Quantization address both at once. They are interpretable by construction because they make the decision by comparing the input to the prototypical example; and their hypothesis margin yields a closed form robustness certificate. However, on high-dimensional data, these shallow models need deep embedding networks to achieve high accuracy, and this voids the robustness certificate. The margin is a statement about distances in the embedding space, and an unconstrained network can map an arbitrarily small input perturbation to arbitrarily large attacks in the embedding space. We close this gap with a theorem: if the embedding network is locally L-Lipschitz around a point, and the classifier is ε-robust around its embedding, then the full model is ε/L-robust in the input space. Based on this, we propose a Lipschitz training scheme to achieve L-Lipschitz properties for embedding networks. We evaluate three classification heads (linear, linear Support Vector Machine, and GLVQ) crossed with two embedding networks (standard and Lipschitz-constrained) on MNIST, CIFAR-10, and Imagenette-160, under FGS, PGD and C&W attacks. The evaluation results show that combining a prototype-based head with a Lipschitz-constrained embedding provides the strongest robustness. This suggests that the Lipschitz constraint provides an additional robustness benefit in addition to the advantage provided by the prototype-based head. Since the Lipschitz constraint is enforced by sampling rather than by construction, the reported robustness is empirical rather than certified.

Introduction

Machine learning systems have achieved remarkable breakthroughs in recent years, enabling applications across a wide range of domains, including medical imaging, computer vision, autonomous driving, and security monitoring. In these safety-critical applications, erroneous predictions or decisions can have potentially catastrophic consequences [Zhou et al.(2022)Zhou, Liu, Ye, Zhu, Zhou, and Yu]. Therefore, such applications demand not only accurate predictions but robustness and interpretability [5 )Göpfert, Artelt, Wersing, and Hammer] ()Carlini, Athalye, Papernot, Brendel, Rauber, Tsipras, Goodfellow, Madry, and Kurakin, Göpfert et al. 2020b); robust in the sense that decisions should remain stable under small changes in the input that a human would consider insignificant [45 )Xu, Ma, Liu, Deb, Liu, Tang, and Jain] ()Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus, Goodfellow et al. 2014; )Goodfellow, Shlens, and Szegedy, Xu et al. 2020); and interpretable in the sense that users should be able to understand why a particular decision was made [Gunning et al.(2019)Gunning, Stefik, Choi, Miller, Stumpf, and Yang, Xu et al.(2019)Xu, Uszkoreit, Du, Fan, Zhao, and Zhu, Biehl et al.(2016)Biehl, Hammer, and Villmann], to assess the system before deployment and to diagnose potential failures afterwards. High accuracy on a test set alone does not guarantee either of these properties.

Szegedy et al. [45 )Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus] were among the first to demonstrate this gap. They showed that small, often imperceptible changes to the input can completely change a network’s prediction, and that such changes often transfer between independently trained models. Over the past decade, researchers have confirmed that even models with excellent accuracy on clean data can be reduced to near-zero accuracy under white-box attacks [31] (Goodfellow et al. 2014; )Goodfellow, Shlens, and Szegedy, Madry et al. 2017). Vulnerability to adversarial attacks can also indicate a lack of interpretability, as the model may rely on features that are not meaningful or recognizable to humans. Therefore, robustness and interpretability are closely connected rather than being distinct goals. Both assess whether the model makes decisions based on features that humans can understand and recognize. This connection is the main focus of this paper.

Two main defense approaches (adversarial training and Lipschitz-based methods) have been proposed in the literature, each with its own strengths and drawbacks. Adversarial training adds adversarial examples into the training data and remains the most effective empirical defense [31 )Zhang, Yu, Jiao, Xing, El Ghaoui, and Jordan] ()Madry, Makelov, Schmidt, Tsipras, and Vladu, Zhang et al. 2019). However, this approach requires an inner optimization at every training step, trades clean accuracy for robustness [46 )Zhang, Yu, Jiao, Xing, El Ghaoui, and Jordan] ()Tsipras, Santurkar, Engstrom, Turner, and Madry, Zhang et al. 2019), and provides no formal guarantees: robustness is only certified against the attacks that were actually tested. Lipschitz-based methods, on the other hand, bound how much the model can amplify input differences, providing an attack-independent guarantee [22 )Hoffman, Roberts, and Yaidal] (Fazlyab et al. 2019; )Fazlyab, Robey, Hassani, Morari, and Pappas, Virmaux and Scaman 2018). Yet a Lipschitz bound alone is not a defense. In fact, the most Lipschitz-stable model is the constant function, which is useless for prediction [59]. Therefore, we propose to combine a Lipschitz constraint for a deep embedding network with the robustness guarantees of shallow classification models.

Prototype-based classifiers are such intrinsically robust classification models—and they additionally offer interpretability as a bonus. A nearest prototype classifier assigns an input to the class of its closest prototype. Thus, its decisions are interpretable as long as the prototypes themselves are interpretable [2 )Davoodi, Mohammadi-zehsamakosh, and Komeili] ()Biehl, Hammer, and Villmann, Davoodi et al. 2023). Since the decision depends solely on distances to a fixed set of prototypes, the distance to the decision boundary can be calculated in closed form. This distance, known as the hypothesis margin [8 )Crammer, Gilad-bachrach, Navot, and Tishby], was originally introduced to derive generalization bounds for Learning Vector Quantization (LVQ)—a specific family of prototype-based classifiers [8 )Hammer, Strickert, and Villmann] ()Crammer, Gilad-bachrach, Navot, and Tishby, Hammer et al. 2005). Saralajew et al. [Saralajew et al.(2020b)Saralajew, Holdijk, and Villmann] later showed that it also yields an adversarial robustness guarantee. Subsequent work has tightened these bounds further and has shown that prototype classifiers can be competitive with certified neural networks [49], while empirical studies report that GLVQ models are robust even before any explicit certification [Saralajew et al.(2020a)Saralajew, Holdijk, Rees, and Villmann].

The challenge, however, is that LVQ operates on distances in the input space, where the Euclidean distance is not meaningful for image data [Khalid and Paaßen()]. The common solution is a hybrid approach: train a deep network ϕ\phi that embeds inputs nonlinearly, and then use prototype-based decisions in this embedding space [7 )Snell, Swersky, and Zemel] ()Chen, Li, Tao, Barnett, Rudin, and Su, Li et al. 2018; )Li, Liu, Chen, and Rudin, Snell et al. 2017). While this approach improves accuracy, it implicitly compromises the formal robustness guarantee. The certificate of a nearest prototype classifier is only valid in the space where the prototypes live. However, an unconstrained embedding network can map an arbitrarily small input change to an arbitrarily large change in the embedding space [26]. Therefore, a certified radius in the embedding space does not provide information regarding robustness in the original input space—unless we have a guarantee that distances in the embedding space are bounded by distances in the input space, which is precisely what a Lipschitz constraint does. As illustrated in Figure 1 (compare the top-left and top-right panels), without a Lipschitz constraint, small changes in the input space can result in large changes in the embedding space, invalidating the robustness guarantee.

Figure 1. Top Left: An adversarial attack of size dd from point xx. Top right: In embedding space, the attack is enlarged to size DD, violating Lipschitz continuity. Bottom: With Lipschitz training, the classification boundary shifts such that the attack fails because DD is now upper-bounded by L⋅dL \cdot d.

Our approach is a combination of prototype-based classification and Lipschitz bounds to achieve better adversarial robustness for deep prototype classifiers. In more detail, our main contributions are:

  • A composition theorem (Theorem 1) that transfers local ϵ\epsilon-robustness guarantees in the embedding space to the input space using LL-Lipschitz continuity of the deep embedding network, yielding a radius of ϵ/L\epsilon/L around each correctly classified point.

  • A Lipschitz loss and corresponding training scheme which enhances the Lipschitz continuity of deep embedding networks by sampling Gaussian noise around the training data during training and punishes violations of the desired Lipschitz constant LL. Because sampling Gaussian noise is cheap, this is substantially cheaper than adversarial training [31 )Madry, Makelov, Schmidt, Tsipras, and Vladu] and less restrictive than a global Lipschitz bound.

  • An evaluation on three data sets (MNIST, CIFAR-10, and Imagenette-160) using three classifiers (linear, linear SVM, and GLVQ) and two embedding networks (standard and Lipschitz-constrained) under three adversarial attacks (FGSM, PGD, and C&W),

This paper is an extension of the conference contribution ESANN [26]. The present paper extends it in four ways: evaluation on three benchmarks instead of one, a sensitivity analysis over the Lipschitz training hyperparameters, the addition of the Carlini–Wagner (C&W) attacks and Projected Gradient Descent (PGD) attacks, and a measurement of residual constraint-violation rates that qualifies the transferred radius as empirical.

Background and Related Work

This section reviews the prior work on which our contribution builds, focusing on two properties of machine learning models: adversarial robustness – the stability of a model’s prediction under small input perturbations [45 )Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus] – and interpretability – the extent to which a human can understand the model’s decision process [3].

Adversarial perturbations, defined as imperceptible changes to the input that flip a model’s prediction, reveal both a security flaw and a fundamental failure of interpretability, as the model’s internal representations diverge from human perception [52 )Göpfert, Artelt, Wersing, and Hammer] ()Xu, Ma, Liu, Deb, Liu, Tang, and Jain, Zhou et al. 2022; )Zhou, Liu, Ye, Zhu, Zhou, and Yu, Göpfert et al. 2020b). Even systems that achieve state-of-the-art accuracy can be successfully attacked, suffering substantial drops in performance [5 )Ilievski, Hammer, van Harmelen, Paassen, Saralajew, Schmid, Biehl, Bolognesi, Dong, Gashteovski, Hitzler, Marra, Minervini, Mundt, Ngomo, Oltramari, Pasi, Saribatur, Serafini, Shawe-Taylor, Shwartz, Skitalinskaya, Stachl, van de Ven, and Villmann] ()Carlini, Athalye, Papernot, Brendel, Rauber, Tsipras, Goodfellow, Madry, and Kurakin, Ilievski et al. 2025).

This phenomenon was first systematically studied by Szegedy et al. who demonstrated that small, often imperceptible perturbations can cause misclassification and may transfer between independently trained models [45 )Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus]. Formally, given a classifier ff with parameters θ\theta, an input-label pair (x,y)(x, y), and a loss function L\mathcal{L}, an adversarial example is found by solving the inner maximization problem [31 )Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus] ()Madry, Makelov, Schmidt, Tsipras, and Vladu, Szegedy et al. 2013):

max⁡η:∥η∥p≤ϵL(θ,x+η,y)(1)\max_{\eta:\lVert\eta\rVert_p\le\epsilon} \mathcal{L}(\theta, x+\eta, y) \tag*{(1)}

where η\eta is an adversarial perturbation, constrained to an ℓp\ell_p-norm ball of radius ϵ\epsilon [Goodfellow et al.(2014)Goodfellow, Shlens, and Szegedy, Madry et al.(2017)Madry, Makelov, Schmidt, Tsipras, and Vladu]. Such perturbations are commonly constrained using ℓ∞\ell_\infty, ℓ2\ell_2, or ℓ0\ell_0 norms [4]. Based on the level of access to the target model, attacks are typically categorized as white-box, where the adversary has full access to the model parameters and gradients, or black-box, where attacks rely on transferability without direct access [Yuan et al.(2019)Yuan, He, Zhu, and Li, Chakraborty et al.(2021)Chakraborty, Alam, Dey, Chattopadhyay, and Mukhopadhyay, Papernot et al.(2017)Papernot, McDaniel, Goodfellow, Jha, Celik, and Swami]. Since white-box attacks provide direct access to the model’s vulnerabilities [Yuan et al.(2019)Yuan, He, Zhu, and Li], they represent the most challenging threat scenario and are the focus of rigorous robustness evaluation.

Among white-box attacks, the fast gradient sign method (FGSM) [Goodfellow et al.(2014)Goodfellow, Shlens, and Szegedy] is a single-step ℓ∞\ell_\infty attack that perturbs the input by ϵ\epsilon in the direction of the sign of the loss gradient. It is computationally efficient and serves as a lower bound on attack strength; a model that fails against FGSM cannot be considered robust in any meaningful sense. Projected gradient descent (PGD) [31 )Madry, Makelov, Schmidt, Tsipras, and Vladu] extends this to a multi-step first-order adversary and is substantially stronger than single-step variants [31 )Shafahi, Najibi, Ghiasi, Xu, Dickerson, Studer, Davis, Taylor, and Goldstein] ()Madry, Makelov, Schmidt, Tsipras, and Vladu, Dong et al. 2018; )Dong, Liao, Pang, Su, Zhu, Hu, and Li, Shafahi et al. 2019). Madry et al. [Madry et al. [31] argue that PGD with random restarts approximates the strongest first-order adversary, establishing it as the standard reference attack for ℓ∞\ell_\infty robustness. Beyond FGSM and PGD, the Carlini & Wagner (C&W) attack [4] formulates adversarial example generation as an optimization problem that jointly minimizes the perturbation norm and a surrogate of the classification margin under ℓ0\ell_0, ℓ2\ell_2, or ℓ∞\ell_\infty constraints. This attack achieves 100% success on both distilled and undistilled networks, revealing that defensive distillation, previously claimed to reduce attack success to 0.5%, provides little real security benefit [4]. Other notable attacks include DeepFool [33 )Moosavi-Dezfooli, Fawzi, and Frossard], which searches for the minimal perturbation crossing the decision boundary by iteratively linearizing it; saliency-guided attacks such as JSMA, which target ℓ0\ell_0 sparsity [Papernot et al.(2016)Papernot, McDaniel, Jha, Fredrikson, Celik, and Swami]; and AutoAttack [9], a parameter-free ensemble of four complementary attacks designed for reliable robustness evaluation. Croce and Hein [9] showed that published defenses frequently report inflated robustness: across 49 deterministic defenses, AutoAttack obtained lower robust accuracy than originally reported in all but one case, with drops exceeding 10 percentage points in 13 cases and by more than 30 percentage points in eight [9].

The design of effective defense mechanisms against adversarial perturbations must navigate a trade-off: improving robustness against adversarial perturbations often reduces accuracy on clean data [Zhang et al.(2019)Zhang, Yu, Jiao, Xing, El Ghaoui, and Jordan]. Moreover, training robust classifiers is computationally expensive and does not provide formal guarantees. Despite these costs, robustly trained models learn feature representations that are more closely aligned with human-perceptible features [46 )Tsipras, Santurkar, Engstrom, Turner, and Madry], suggesting that robustness and interpretability are interrelated.

To enhance adversarial robustness, numerous defense strategies have been proposed, ranging from adversarial training [Goodfellow et al.(2014)Goodfellow, Shlens, and Szegedy, Madry et al.(2017)Madry, Makelov, Schmidt, Tsipras, and Vladu] to regularization techniques [36]. Adversarial training, which augments the training set with adversarial examples, has emerged as one of the most effective empirical defenses. Zhang et al. [Zhang et al.(2019)Zhang, Yu, Jiao, Xing, El Ghaoui, and Jordan] formalized the robustness–accuracy trade-off through TRADES, which achieves a theoretically principled balance between standard accuracy and adversarial robustness, winning first place in the NeurIPS 2018 Adversarial Vision Challenge and surpassing the runner-up by 11.41% in mean ℓ2\ell_2 perturbation distance. However, adversarial training is computationally expensive and does not provide formal guarantees.

Lipschitz theory instead offers guarantees that hold independently of any particular attack [59]: the Lipschitz constant LL upper-bounds the amplification of input differences and is thus a direct measure of perturbation sensitivity [Fazlyab et al.(2019)Fazlyab, Robey, Hassani, Morari, and Pappas, Virmaux and Scaman(2018)]. This motivates constraining LL during training via Jacobian or spectral norm regularization [Zhao et al.(2021)Zhao, Zhang, Zhang, Wu, Jin, Zhou, Jin, Dou, and Yan, Yoshida and Miyato(2017)], input-gradient regularization [15], or weight-matrix conditioning [Feng et al.(2025)Feng, Lin, Gao, and Wei]. For instance, Hoffman et al. [22 )Hoffman, Roberts, and Yaida] approximate local Lipschitz constraints by penalizing the Jacobian norm via random projection (a computationally efficient estimator of local sensitivity), without generating adversarial perturbations. The framework has two well-known limitations. First, the most Lipschitz-stable model is the constant function with L=0L = 0, which is useless for prediction [59]. Second, computing LL exactly is NP-hard even for two-layer networks [48]; consequently, all practical estimators must trade tightness against scalability [Fazlyab et al.(2019)Fazlyab, Robey, Hassani, Morari, and Pappas]. Furthermore, Ross and Doshi-Velez [36] found that penalizing input-gradient norms improves robustness to transferred adversarial examples while simultaneously making input gradients more interpretable as explanations. Specifically, human evaluators rated the resulting misclassifications as more plausible than those of other defended models, suggesting a strong fundamental connection between robustness and interpretability [47 )Vadillo, Santana, and Lozano]. Some intrinsically interpretable models also provide robustness guarantees. Prototype-based models perform classification by comparing input data points to prototypical examples for each class, which makes them ininterpretable by construction: as long as the prototypes themselves are interpretable, so are the decisions [2 )Davoodi, Mohammadizadehsamakosh, and Komeili] ()Biehl, Hammer, and Villmann, Davoodi et al. 2023). Formally, a nearest prototype classifier assigns an input data point xx to the label of the closest prototype ww from a learned, finite set of prototypes W⊂X\mathcal{W} \subset\mathcal{X} under a dissimilarity d:X×X→R≥0d : \mathcal{X} \times\mathcal{X} \to\mathbb{R}_{\geq0} [Saralajew et al.(2020b)Saralajew, Holdijk, and Villmann]. Learning Vector Quantization (LVQ) [27] is the foundational nearest prototype classifier; Generalized LVQ (GLVQ) [40] replaces the heuristic update with a differentiable cost function, and further extensions either learn the metric itself [20 )Schneider, Biehl, and Hammer] (), Schneider et al. 2009) or introduce probabilistic variants [42]. The connection between prototype models and robustness is structural rather than incidental. Crammer et al. [8 )Crammer, Gilad-bachrach, Navot, and Tishby] showed that LVQ-type learning maximizes a hypothesis margin, and Hammer et al. [Hammer et al.(2005)Hammer, Strickert, and Villmann] derived corresponding generalization bounds that depend on this margin rather than on input dimensionality. Because the decision of a nearest prototype classifier depends only on distances to a finite prototype set, the distance from a point to the decision boundary can be computed in closed form, yielding a robustness certificate rather than an empirical estimate. Saralajew et al. [Saralajew et al.(2020b)Saralajew, Holdijk, and Villmann] made this explicit, showing that the hypothesis margin of nearest prototype classifiers is a tight lower bound on adversarial perturbation size and can be computed in constant time for seminorm-induced dissimilarities. GLVQ trained with triplet losses automatically maximizes this margin, certifying an entire test set in roughly the time required by randomized smoothing for a single CIFAR-10 sample. Voráček and Hein [49] extended this analysis, demonstrating that ℓ2\ell_2-PNPC (Provably Adversarially Robust Nearest Prototype Classifier) achieves the best certified ℓ2\ell_2-robust accuracy on MNIST among deterministic certifiers, outperforming Saralajew et al.’s ℓ2\ell_2-GLVQ by approximately 6% robust accuracy at ϵ2=1.58\epsilon_2 = 1.58. On CIFAR-10 under the LPIPS perceptual metric, their Perceptual-PNPC reaches 80.3% clean accuracy with 40.5% certified robust accuracy, compared to 71.6% clean and only 9.8% empirical robust accuracy for an adversarially trained ResNet-50 baseline. Empirically, Saralajew et al. [Saralajew et al.(2020a)Saralajew, Holdijk, Rees, and Villmann] found that GLVQ and its tangent-distance variant GTLVQ exhibit high base robustness without any adversarial training, and that adversarial examples against these models resemble semantically meaningful interpolations towards the target class rather than arbitrary noise.

This connection between margin and robustness is not unique to prototype-based classifiers. The linear Support Vector Machine (SVM) is a classic maximum-margin classifier, and its robustness properties have been extensively studied [32 )Dakhale, Satwik, Kumar, Narayana, Bhurane, and Kothari] (), Dakhale et al. 2023). Inspired by these results, Elsayed et al. [32] proposed a margin-maximization objective for deep neural networks, demonstrating that it enhances robustness to adversarial perturbations by promoting a larger geometric margin. This approach effectively extends the classical SVM margin-maximization principle to the deep learning setting. Yu et al. [Yu et al.(2019)Yu, Amizadeh, Kim, Pagnoni, Chun, Weimer, and Interlandi] further showed that classical machine learning pipelines, including SVMs, can be integrated into neural network architectures and jointly optimized through backpropagation. Their findings highlight the flexibility of SVM-based approaches within modern deep learning frameworks. Motivated by this, we include a linear SVM as an additional baseline alongside linear and GLVQ classifiers, enabling a direct comparison between a conventional maximum-margin classifier and the prototype-based GLVQ approach. Related studies investigating margin-based robustness in non-parametric and nearest-neighbor classifiers have reported similar findings [50 )Wang, Jha,] and Chaudhuri, Yang et al. [53]Yang, Rashtchian, Wang, and Chaudhuri].

Unfortunately, shallow classifiers such as GLVQ or SVM tend to struggle in highly nonlinear settings such as image data, where the Euclidean distance in input space is uninformative for class discrimination [Khalid and Paassen()], a phenomenon consistent with the accuracy–interpretability trade-off described in the XAI literature [Gunning et al.(2019)Gunning, Stefik, Choi, Miller, Stumpf, and Yang, Xu et al.(2019)Xu, Uszkoreit, Du, Fan, Zhao, and Zhu]. This has motivated hybrid architectures that use a deep neural network to perform a nonlinear embedding while retaining prototype-based decisions in the embedding space. Examples are Prototypical Networks [44 )Snell, Swersky, and Zemel], autoencoder-plus-prototype-layer models whose prototypes decode back to the input space [30 )Li, Liu, Chen, and Rudin], ProtoPNet and its part-prototype descendants [7 )Chen, Li, Tao, Barnett, Rudin, and Su], classification-by-components [Saralajew et al.(2019)Saralajew, Holdijk, Rees, Asan, and Villmann], and prototype–rule hybrids [1]. These hybrid models improve classification performance but lose robustness guarantees, because the certificate of a nearest prototype classifier is only valid in the space where the prototypes are defined: the embedding space [26, 2 )Biehl, Hammer, and Villmann]. An unconstrained embedding network can map an arbitrarily small input perturbation to an arbitrarily large embedding displacement, so a certified radius in Φ\Phi provides no information regarding robustness in X\mathcal{X} unless the embedding network’s sensitivity is itself bounded. Bounding the embedding network’s sensitivity is, therefore, the starting point for our method.

Methodology

Our goal is to leverage the interpretability and robustness advantages of prototype-based models when combined with deep embedding networks for pre-processing. We first establish the theoretical foundation of our method before describing our proposed algorithm for Lipschitz training.

Adversarial Robustness

Assume a deep neural network ϕ:X→Φ\phi:\mathcal{X}\to\Phi, which maps input data from some input space X\mathcal{X} to an nn-dimensional embedding space Φ\Phi. Further, assume that the input space X\mathcal{X} is equipped with some metric dd and the embedding space Φ\Phi with some metric DD. Finally, assume that we have trained a classifier f:Φ→{1,…,K}f:\Phi\to\{1,\ldots,K\} which maps data in the embedding space to one of KK labels.

The core challenge we try to address is that a deep neural network ϕ\phi may yield an embedding space that improves classification accuracy but breaks adversarial robustness. Points that are close in the input space may be suddenly far apart in the embedding space. Hence, even if the classifier ff is adversarially robust in the embedding space, this does not guarantee adversarial robustness in the input space (see Figure 1, top). Our core argument is straightforward: Lipschitz-continuity limits how much distances in the input space can be stretched in the embedding space (see Figure 1, bottom). Hence, a robust classifier in the embedding space, in combination with a deep network that satisfies the Lipschitz property, yields a robust classifier in the input space.

To make our argument precise, we first introduce a formal definition of (local) adversarial robustness.

Definition 1. Let Φ\Phi be some set, let ff be some classifier f:Φ→{1,…,K}f:\Phi\to\{1,\ldots,K\} for some K∈NK\in\mathbb{N}, and let D:Φ×Φ→RD:\Phi\times\Phi\to\mathbb{R} be a metric on Φ\Phi. Further, let x∈Φx\in\Phi and ϵ∈R\epsilon\in\mathbb{R} with ϵ>0\epsilon>0. We say

that ff is locally ϵ\epsilon-robust around xx (with respect to metric DD) if and only if for all y∈Φy \in\Phi with D(x,y)≤ϵD(x,y) \le\epsilon we have f(x)=f(y)f(x) = f(y).

For the purpose of our argument, we assume that this property holds point-wise for all correctly classified points, in particular for the held-out points used for robustness evaluation. It is important to note that the robustness radius ϵ\epsilon is not a fixed constant but depends on the input point xx. For a nearest prototype classifier such as GLVQ, the certified radius around a correctly classified point xx is half the distance from xx to the decision boundary [8] induced by the closest correct and closest incorrect prototype. Let d(x,w)=∥x−w∥d(x,w) = \lVert x-w \rVert denote the ℓ2\ell_2 distance between xx and ww. Saralajew et al. [Saralajew et al. (2020b)] show that the hypothesis margin at xx can be obtained in constant time during the inference step as:

ϵ(x)=12(∥ϕ(x)−ϕ(w−)∥−∥ϕ(x)−ϕ(w+)∥),(2)\epsilon(x) = \frac{1}{2}\left(\lVert\phi(x)-\phi(w^{-}) \rVert- \lVert\phi(x)-\phi(w^{+}) \rVert\right), \tag*{(2)}

where w+w^{+} and w−w^{-} denote the closest prototype of the correct class and any incorrect class, respectively. This quantity lower-bounds the sample margin and hence the norm of any adversarial perturbation that changes the predicted label, with the bound being tight. The notion originates with Crammer et al. [Crammer et al. [8], who introduced the hypothesis margin to derive generalization bounds for LVQ under the ℓ2\ell_2 norm. The certificate applies only to correctly classified points; for misclassified points, the signed margin of [Saralajew et al. (2020b)] is negative, and no radius is guaranteed.

Despite the robustness certificates of LVQ models, it is easy to construct a case where an LVQ model loses its robustness when combined with a deep embedding model. The certificate in Eq. (2) is a statement about distances in the embedding space Φ\Phi. If ϕ\phi is unconstrained, an input perturbation of arbitrarily small norm in X\mathcal{X} can be mapped to a distance larger than ϵ(x)\epsilon(x) in Φ\Phi, so the robustness guarantee in Φ\Phi, by itself, carries no information about robustness in X\mathcal{X}. Therefore, we also need Lipschitz continuity.

Lipschitz continuity

To ensure that robustness guarantees in the embedding space translate to robustness guarantees for a deep prototype classifier (including a deep embedding model), we require (local) Lipschitz continuity as a second ingredient.

Definition 2. Let X\mathcal{X} and Φ\Phi be sets, let ϕ:X→Φ\phi: \mathcal{X} \to\Phi, let d:X×X→Rd : \mathcal{X} \times\mathcal{X} \to\mathbb{R} be a metric on X\mathcal{X}, and let D:Φ×Φ→RD : \Phi\times\Phi\to\mathbb{R} be a metric on Φ\Phi. Further, let x∈Xx \in\mathcal{X} and let L∈RL \in\mathbb{R} with L>0L > 0. We say that ϕ\phi is locally LL-Lipschitz-continuous around xx (with respect to metrics dd and DD) if and only if for all y∈Xy \in\mathcal{X} we obtain

D(ϕ(x),ϕ(y))≤L⋅d(x,y).(3)D(\phi(x),\phi(y)) \leq L \cdot d(x,y). \tag*{(3)}

It follows that the local robustness of a classifier ff, combined with the local Lipschitz continuity of an embedding model ϕ\phi, implies the local robustness of ff combined with ϕ\phi.

Theorem 1. Let X\mathcal{X} and Φ\Phi be sets, let ϕ:X→Φ\phi: \mathcal{X} \to\Phi, let f:Φ→{1,…,K}f : \Phi\to\{1,\ldots,K\} for some K∈NK \in\mathbb{N}, let d:X×X→Rd : \mathcal{X} \times\mathcal{X} \to\mathbb{R} be a metric on X\mathcal{X}, and let D:Φ×Φ→RD : \Phi\times\Phi\to\mathbb{R} be a metric on Φ\Phi. Further, let x∈Xx \in\mathcal{X}, and ϵ,L∈R\epsilon, L \in\mathbb{R} with ϵ>0\epsilon> 0 and L>0L > 0. Then it holds: if ϕ\phi is locally LL-Lipschitz around xx (w.r.t. metrics dd and DD) and ff is locally ϵ\epsilon-robust around ϕ(x)\phi(x) (w.r.t. metric DD), then ϕ∘f\phi\circ f (defined as f(ϕ(x))f(\phi(x))) is ϵ\epsilon-robust around xx with respect to dd.

Proof. Consider any y∈Xy \in\mathcal{X} with d(x,y)≤ϵLd(x,y) \leq\frac{\epsilon}{L}. Then, D(ϕ(x),ϕ(y))≤L⋅d(x,y)≤ϵD(\phi(x),\phi(y)) \leq L \cdot d(x,y) \leq\epsilon because ϕ\phi is LL-Lipschitz around xx with respect to dd and DD. Hence, f(ϕ(x))=f(ϕ(y))f(\phi(x)) = f(\phi(y)) because ff is ϵ\epsilon-robust around ϕ(x)\phi(x) w.r.t. DD.

Here dd and DD are the ℓ2\ell_2 metrics on X\mathcal{X} and Φ\Phi; the squaring in Equation (4) applies inside the Lipschitz loss only, so that larger violations are punished more strongly.

While this theorem is, admittedly, somewhat trivial, it is highly useful for our purpose. The theorem implies that we can leverage the robustness and interpretability properties of GLVQ models in combination with deep networks if the deep networks are Lipschitz continuous with sufficiently small LL. Hence, we devise a training method that encourages this Lipschitz property.

Lipschitz Training

Our goal is to construct a training scheme which ensures that a deep neural network ϕ\phi is locally LL-Lipschitz-continuous for some desired LL around all training data points xx. To achieve this goal, our core strategy is quite simple: For each training data point xix_i, we sample one neighbor yiy_i by adding Gaussian random noise to xix_i. Then, we punish violations of the Lipschitz property by imposing the loss following loss:

LLipschitz:=∑i=1N[D(ϕ(xi),ϕ(yi))2−L2⋅d(xi,yi)2]+,(4)\mathcal{L}_{\mathrm{Lipschitz}} := \sum_{i=1}^{N} \left[D(\phi(x_i),\phi(y_i))^2 - L^2 \cdot d(x_i,y_i)^2\right]_{+}, \tag*{(4)}

where,

yi∼N(xi,σ2⋅I)(5)y_i \sim\mathcal{N}(x_i,\sigma^2 \cdot I) \tag*{(5)}

and where [⋅]+[\cdot]_{+} denotes the hinge loss/the rectified linear unit [ℓ]+=max⁡{0,ℓ}[\ell]_{+} = \max\{0,\ell\} and N\mathcal{N} denotes the multivariate Gaussian density.

For training, the Lipschitz loss (4) is added to the regular classification loss of the classifier. In our case, we train a deep embedding network ϕ\phi with a final linear layer for classification using the cross-entropy loss

L=LCE+λLLipschitz,(6)\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda\mathcal{L}_{\mathrm{Lipschitz}}, \tag*{(6)}

where λ\lambda is the Lipschitz weight, controlling the strength of the Lipschitz loss relative to the cross-entropy loss.

The loss is then minimized via standard deep learning optimizers, such as Adam.

Hyperparameters: Lipschitz training depends on three hyperparameters, each playing a distinct role.

The hyperparameter LL is the desired upper bound on the Lipschitz constant of the embedding network ϕ\phi. As established in Theorem 1, if ϕ\phi is LL-Lipschitz and the classifier ff is ϵ\epsilon-robust in the embedding space, then the composed classifier is ϵ/L\epsilon/L-robust in the input space. Thus, LL enters the robustness radius directly through the factor 1/L1/L. It may, therefore, appear tempting to set LL as low as possible to achieve the best possible robustness bound. However, imposing tighter Lipschitz continuity on ϕ\phi also means that distances within the embedding space shrink, which in turn reduces the distances from which the hypothesis margin in Equation (2) is computed. The radius ϵ(x)/L\epsilon(x)/L that Theorem 1 transfers to the input space is therefore a ratio of two quantities that both shrink as LL shrinks, so it does not simply grow as 1/L1/L. Conversely, a larger LL relaxes the constraint, potentially improving accuracy but weakening the robustness guarantee. The choice of LL should therefore be guided by the desired perturbation budget in the input space.

The second hyperparameter is the Lipschitz weight λ\lambda, which controls how strongly violations are penalized relative to the classification loss. The higher λ\lambda is, the more the Lipschitz loss is weighted and the fewer violations of the Lipschitz loss are expected. However, higher λ\lambda may also hurt classification accuracy (especially on non-attacked data) because the classification loss has less influence on training. λ\lambda can thus be used to navigate the accuracy-robustness tradeoff.

The third hyperparameter is the standard deviation σ\sigma of the isotropic, zero-mean Gaussian noise from which the neighbor yiy_i for each training sample xix_i is generated. Lower values mean that neighbors are samples more closely to the training data, higher values mean that neighbors are farther away. We generally recommend to base σ\sigma on the expected strength of adversarial attacks.

All three hyperparameters are also investigated experimentally and discussed in § 4.4.

Computational Efficiency: In terms of computational efficiency, Lipschitz training is releatively cheap. For every training point xix_i in a mini-batch, we sample a single neighbor yiy_i via Gaussian noise. Thus, the asymptotic training efficiency remains unchanged. Lipschitz training compares particularly favorably to adversarial training [31 )Madry, Makelov, Schmidt, Tsipras, and Vladu], where yiy_i needs to be generated via projected gradient descent, which requires an inner maximization involving several forward and backward passes for every single training data point during training.

Lack of theoretical guarantees: We acknowledge that Lipschitz training does not guarantee a certain Lipschitz constant but only punishes violations of the desired constant LL for the distance between a training sample xix_i and the sampled neighbor yiy_i. If the Lipschitz loss (4) becomes zero during training, LL does hold for the training data because then ∥ϕ(xi)−ϕ(yi)∥2≤L2∥xi−yi∥2\lVert\phi(x_i)-\phi(y_i)\rVert^2 \le L^2\lVert x_i-y_i\rVert^2 for all (xi,yi)(x_i,y_i) pairs. Further, we recommend to re-sample the neighbors yiy_i in each mini-batch, thus providing a more comprehensive stochastic sampling in each direction. If training would continue indefinitely under infinitely strong weight λ\lambda, local LL-Lipschitz continuity would eventually result. In practice, though, the effective Lipschitz constant of ϕ\phi will be higher than LL. We investigate the empiric effectiveness of Lipschitz training in much more detail in the next section.

Classification Heads

After training the embedding network ϕ\phi, we use one of three classification heads for inference. By “classification head”, we mean the final layer ff that maps the embedding ϕ(x)\phi(x) to a class prediction.

Linear: The standard linear+softmax layer with cross-entropy loss serves as our baseline and is used during the first training stage.

Linear SVM: We extract the embeddings of the training fold with the embedding network frozen and fit a linear SVM on these fixed feature vectors using scikit-learn’s LinearSVC (C=1C = 1, one-vs-rest). Since the decision function of a linear SVM, f(z)=Wz+bf(z) = Wz + b, has the same form as a linear layer, the learned WW and bb are copied into a linear layer on top of the frozen embedding network [Yu et al. (2019)Yu, Amizadeh, Kim, Pagnoni, Chun, Weimer, and Interlandi], so that the same gradient-based attacks apply as for the other heads. This head provides a maximum-margin classifier in the embedding space but lacks interpretability.

Generalized Learning Vector Quantization (GLVQ): A prototype-based model to achieve interpretable classification. The prototypes w1,…,wKw_1, \ldots, w_K are initialized in the input space and then mapped into Φ\Phi by the embedding network ϕ\phi, rather than being free vectors in Φ\Phi. This keeps them interpretable as images throughout training and guarantees that they lie on the image of ϕ\phi, which makes the distances in embedding space and input space comparable between prototypes and data. The optimization of the prototypes occurs via Adam on the GLVQ loss [40]. However, in contrast to regular GLVQ, because the prototypes live in X\mathcal{X}, the GLVQ loss needs to be propagated through the embedding network ϕ\phi. Still, the parameters of ϕ\phi remain frozen at this stage.

Because both GLVQ and SVM are maximum margin classifiers, we assume that they are locally ϵ\epsilon-robust, such that the Lipschitz-continuous embedding network ϕ\phi combined with GLVQ or SVM should be ϵL\frac{\epsilon}{L}-robust, following Theorem 1. We will test this hypothesis experimentally.

Summary of Training Procedure

In summary, training proceeds in two stages. Stage 1: Lipschitz Embedding Training. We train the embedding network ϕ\phi jointly with a linear (+softmax) classification head, minimizing Eq. (6). This step (hopefully) yields a locally LL-Lipschitz continuous embedding network ϕ\phi.

Stage 2: Freeze and Fit Classifier Heads. After Stage 1, we freeze the embedding network ϕ\phi, keeping it in evaluation mode and setting requires_grad = False. The linear head is not re-trained: we keep the one obtained in Stage 1. Only the SVM and the GLVQ prototypes are fitted, both on top of the frozen embedding, ensuring that all three heads share identical features and differ only in the decision rule.

Experimental Setup and Datasets

Our primary objective is to test whether a Lipschitz-constrained embedding network combined with a maximum-margin classifier (GLVQ or SVM) achieves higher adversarial robustness than all baseline combinations. Specifically, we aim to:

  1. isolate the effect of the Lipschitz loss by comparing standard and Lipschitz-constrained embedding networks with the same classification head;

  1. isolate the effect of the classification head by comparing Linear, SVM, and GLVQ heads on the same frozen embedding; and

  1. demonstrate that the proposed Lipschitz-GLVQ model achieves enhanced robustness compared to all baselines, as measured by standard adversarial attacks.

We evaluate the proposed scheme on three image classification benchmarks of increasing difficulty and resolution. Our aim is not to reach state-of-the-art clean accuracy but to investigate the effect of the Lipschitz loss: within each dataset, every model shares the same embedding network architecture, the same optimization budget, and the same data splits, so that differences in adversarial accuracy can be attributed to the Lipschitz loss and to the choice of classification head rather than to model capacity or tuning.

Datasets

The detailed description of all three datasets is given below:

MNIST

The first benchmark is MNIST [29 )LeCun, Cortes, and Burges], consisting of 28×2828 \times28 gray-scale images (784784 input features) with raw pixel values in [0,255][0, 255]. The images are loaded with transforms.ToTensor(), which maps them to [0,1][0, 1]; no further channel normalization is applied. The torchvision.datasets release contains 70,00070,000 samples (60,00060,000 train, 10,00010,000 test) evenly distributed over ten digit classes (00–99). The class distribution is approximately, but not exactly, uniform.

CIFAR-10

The second benchmark is CIFAR-10 [28] (accessed via PyTorch), described by its authors as “60000 32x32 color images in 10 classes, with 6000 images per class”, split into 50,00050,000 training and 10,00010,000 test images. The images are normalized with the channel mean and standard deviation computed on that split.

Imagenette

The third benchmark is Imagenette [23], described by Howard et al. as “a subset of 10 easily classified classes from Imagenet”. We use the imagenette-160 release with its predefined train / val directory structure (Imagenette-1601), and resize all images to 64×6464 \times64 pixels to keep the embedding network architecture consistent with the CIFAR-10 setting. We use the 9,4699,469 images in the train directory, which are normalized with the channel mean and standard deviation computed on that split.

Model Architecture and Feature Extraction

Embedding network: We consider image classification with a convolutional neural network (CNN) for feature extraction. Detailed descriptions for each dataset and its respective settings for the CNN embedding network are provided in Table 1. The same embedding network architecture is used for both the standard (unregularized) models and the Lipschitz-constrained models, ensuring that any differences in performance can be attributed solely to the Lipschitz loss rather than architectural variations. These architectures were chosen to be lightweight and representative of typical CNNs for each dataset. A systematic hyperparameter search over network architectures was outside the scope of this paper, and other architectures may yield better performance.

MNISTCIFAR-10Imagenette-160
Classes101010
Samples used (train)60, 00050, 0009, 469
Native resolution28×28×128 \times28 \times132×32×332 \times32 \times3variable (resized to 64×6464 \times64)
Input to the network28×28×128 \times28 \times132×32×332 \times32 \times364×64×364 \times64 \times3
Pixel range[0,1][0, 1][0,1][0, 1][0,1][0, 1]
Channel normalizationnonesplit mean/stdsplit mean/std
Convolutional layers233
Batch normalizationnoyesyes
ActivationReLUReLUReLU
Pooling2×22 \times2 max2×22 \times2 max2×22 \times2 max
Dropout0.00.50.5
Embedding dim(Φ\Phi)166464

Table 1. Datasets, input pre-processing and network parameters. The pixel range refers to the values after ToTensor and before channel normalization.

Models’ Architecture: We compare six models, obtained by crossing three classification heads (linear, SVM, and GLVQ) with two training schemes for the embedding network (with and without the Lipschitz loss), as listed in Table 2. The training procedure consists of two stages. The embedding network is first trained end-to-end together with a linear (+softmax) head, with the Lipschitz loss of Eq. (6) in the Lipschitz-constrained setting and without it in the baseline setting, and is then frozen, as shown in Figure 2. The SVM and GLVQ heads are subsequently fitted on top of this frozen embedding. Under this design, any difference between Linear and Lipschitz-Linear can be attributed to the Lipschitz loss alone, and any difference between Lipschitz-Linear and Lipschitz-GLVQ is caused by the classification head alone. Consequently, this setup allows us to isolate the effect of the Lipschitz loss from that of the classification head. We expect that the two Lipschitz variants with a margin-based heads (Lipschitz-SVM and Lipschitz-GLVQ) will gain substantial adversarial robustness, as they combine adversarial robustness in the embedding space (due to maximum margin properties) with the Lipschitz continuity of the embedding network. The added value of Lipschitz-GLVQ over Lipschitz-SVM should only lie in additional interpretability due to the prototype-based classification scheme.

ModelEmbedding NetClassification headInterpretable?
LINEARstandardlinear+softmax layer, cross-entropy lossno
SVMstandardlinear SVM (C=1C = 1, one-vs-rest) fitted on the frozen embedding and converted into an equivalent linear layer so that gradient-based attacks applyno
GLVQstandardone prototype per class, GLVQ cost with the sigmoid link; only the prototypes are updatedyes
Lipschitz_LinearLipschitzas LINEARno
Lipschitz_SVMLipschitzas SVMno
Lipschitz_GLVQLipschitzas GLVQyes

Table 2. The six models compared on every dataset. All heads within each configuration share the same frozen embedding network.

Proposed architecture showing the input space, CNN embedding, embedding space, classifier, and predicted class

Figure 2. The proposed architecture where the Lipschitz loss is applied to the embedding network (CNN) ϕ\phi, which is then frozen; after that, the classification head classifies in the embedding space Φ\Phi.

Training Protocol

The following training details apply to all experiments. Models were trained for 50 epochs on all three datasets to ensure that no model receives a larger optimization budget than another. The batch sizes are 32 for MNIST, 128 for CIFAR-10, and 64 for Imagenette-160. For GLVQ, we use one prototype per class, as described in § 3.4. The prototypes are the only parameters updated during GLVQ training: the embedding network is frozen, and the optimizer is constructed over the prototypes only. Prototypes are initialized as one randomly drawn training image per class and are updated directly in input space, so that they remain interpretable as images. The embedding networks and linear heads are trained jointly in Stage 1 with an Adam optimizer (lr =10−3= 10^{-3}, weight decay 10−410^{-4}) for 50 epochs; the GLVQ prototypes are trained with a separate Adam instance at the same learning rate. The SVM is fitted with scikit-learn rather than by gradient descent, as described in § 3.4. The remaining hyper-parameter settings are listed in Table 3.

MNISTCIFAR-10Imagenette
Epochs505050
Batch size3212864
OptimiserAdam, lr = 10−310^{-3}, weight decay 10−410^{-4}Adam, lr = 10−310^{-3}, weight decay 10−410^{-4}Adam, lr = 10−310^{-3}, weight decay 10−410^{-4}
Cross-validation folds555
Prototypes per class PP111
Embedding dimension nn166464
Lipschitz loss
Neighborhood mean μ\mu000
Neighborhood std. dev. σ\sigma0.10.030.03
Lipschitz constant LL0.0320.0320.032
Lipschitz weight λ\lambda101010
Attack parameters
FGSM/PGD budget ϵ\epsilon[0, 0.4], 41 pts[0, 0.0667], 35 pts[0, 0.0667], 35 pts
Evaluation batch size200200200
PGD steps TT402020
PGD random restarts111
C&W binary-search steps NN999
C&W optimisation steps MM100010001000
C&W initial constant111
C&W confidence κ\kappa000

Table 3. Training hyperparameters per dataset.

Lipschitz Hyperparameter Selection

As discussed previously in § 3, Lipschitz training has three hyperparameters: the desired Lipschitz constant LL, the Lipschitz weight λ\lambda, and the neighborhood standard deviation σ\sigma. We selected these values on a held-out portion of the training split (using only the official training split), keeping them fixed across all cross-validation folds.

Lipschitz constant LL. As discussed in § 3, LL has two opposing effects. On the one hand, Theorem 1 certifies a radius of ϵ/L\epsilon/L in input space, so a smaller LL (if actually achieved) means a larger guaranteed radius. On the other hand, a smaller LL constrains the embedding network ϕ\phi more strongly, which may cost clean accuracy. The choice of LL therefore has to be made empirically. We tested L∈{0,0.032,0.071,0.1,0.224,0.316,0.707,0.775,1}L \in\{0, 0.032, 0.071, 0.1, 0.224, 0.316, 0.707, 0.775, 1\} (see Figure 7). Values of L≳0.707L \gtrsim0.707 left the constraint inactive, while L≤0.032L \le0.032 produced no additional robustness gain. We therefore chose L=0.032L = 0.032 as a conservative operating point, balancing robustness with (unattacked) classification accuracy.

Figure 7. Effect of target Lipschitz constant L∈{0,0.0316,0.0707,0.1,0.224,0.316,0.707,0.775,1}L \in\{0, 0.0316, 0.0707, 0.1, 0.224, 0.316, 0.707, 0.775, 1\} at fixed λ\lambda. Larger LL leaves the hinge slack and the Lipschitz loss inactive.

Lipschitz weight λ\lambda. λ\lambda is purely an optimization-side quantity; it weighs the Lipschitz loss versus the classification loss. We tested λ∈{1,5,10,20,50,100}\lambda\in\{1, 5, 10, 20, 50, 100\} (see Figure 6). Higher values reduce the rate of violations against the Lipschitz constraint but cost clean accuracy (i.e. accuracy without adversarial attacks) on CIFAR-10 and Imagenette-160 (Table 5). We selected λ=10\lambda= 10 as the default Lipschitz weight.

Figure 6. Effect of Lipschitz weight λ∈{1,5,10,20,50,100}\lambda\in\{1, 5, 10, 20, 50, 100\} at fixed L=0.032L = 0.032, over 50 epochs. Solid curves: weighted Lipschitz loss (left, log axis). Dashed: classification loss (right axis).

Datasetλ\lambdaViolationMean stretch ratioClean accuracy (%)
MNIST10.8670.04998.47
50.4100.03298.33
100.2520.02898.32
200.1470.02598.13
500.0700.02398.21
1000.0410.02198.08
CIFAR-1011.0000.20078.12
50.9880.10777.17
100.9520.08175.54
200.8390.06173.56
500.5900.04370.17
1000.4010.03467.00
Imagenette-16010.9830.10376.36
50.7700.05472.31
100.5510.04166.49
200.3340.03156.11
500.1890.02640.09
1000.1310.02330.69

Table 5. How strictly the Lipschitz constraint is enforced as the weight λ\lambda varies, at fixed L=0.032L = 0.032. Two quantities measure this on the same sampled training pairs: “violation” counts the fraction of pairs for which the constraint of Eq. (4) is not satisfied, while “mean stretch ratio” reports by how much the embedding is stretched on those pairs. Values are taken at the final training epoch and averaged over 5 folds.

Neighborhood standard deviation σ\sigma. The parameter σ\sigma fixes the radius at which neighbors are sampled during training for testing violations of the Lipschitz condition. To effectively check the Lipschitz constraints for small distances where adversarial attacks are expected, σ\sigma should be of the same order as the largest attack budget ϵmax⁡\epsilon_{\max} evaluated on that dataset. Guided by this, we swept σ∈{0.03,0.07,0.1,0.2,0.3,0.5}\sigma\in\{0.03, 0.07, 0.1, 0.2, 0.3, 0.5\} and selected σ=0.1\sigma= 0.1 for MNIST and σ=0.03\sigma= 0.03 for CIFAR-10 and Imagenette-160, which are the smallest values at which the Lipschitz loss remains clearly active throughout training; § 5.2 discusses this sweep in detail.

Evaluation Protocol

All experiments use 5-fold cross-validation over the training split, with folds generated by a shuffled KFold split under a fixed seed. In each fold, all six models are trained on four folds, and the held-out fold is used for both clean and adversarial evaluation. All results are averaged over the five folds and reported as the mean ±\pm one standard deviation. All hyperparameters were fixed before running any experiments (see § 4.4).

Implementation details

The Lipschitz loss is computed on the same mini-batch as the classification loss, using the pre-logit embedding. As discussed above, the SVM and GLVQ classification heads are fitted on top of the frozen embedding network; therefore, the Lipschitz loss plays no role in their training. Mini-batches are drawn by reshuffling the training fold at the start of every epoch. Perturbed samples used for the Lipschitz loss are not clipped to the valid pixel range, since the local Lipschitz condition is meant to hold in the ambient space and clipping would bias the sampled neighborhood at the boundary of [0,1][0, 1]. In addition to optimization and hyper-parameter tuning, a crucial implementation detail concerns the handling of input normalization. Because the networks operate on channel-normalized inputs while perturbation budgets are only meaningful in raw pixel units, every attack follows the same three-step convention: the image is un-normalized into raw pixel space, the adversarial attack is applied in pixel space, enforcing the attack budget ϵ\epsilon as well as the bounds of the pixel range [0,1][0, 1]. Finally, the result is re-normalized before being fed to the model. As loss functions for adversarial attacks, cross-entropy is used for Linear and SVM classifiers and the GLVQ loss for GLVQ models. For the Lipschitz-GLVQ model, the attack uses the classification part of the cost only, without the Lipschitz loss term, since the Lipschitz loss is not part of the decision rule and including it would provide the attacker with a gradient that does not correspond to the model’s predictions.

In preliminary experiments with L=1L = 1, we mistakenly included the Lipschitz loss in the loss that was attacked. Because the Lipschitz loss directly works against adversarial attacks, this made the adversarial robustness appear much better than it actually was. Therefore, we caution other researchers against this mistake.

Attack schemes and robustness evaluation

We evaluate the adversarial robustness by measuring the accuracy under FGSM [Goodfellow et al.(2014)Goodfellow, Shlens, and Szegedy], PGD [31 )Madry, Makelov, Schmidt, Tsipras, and Vladu], and C&W ℓ2\ell_2 attacks [4]. All three types of attack are computed on the held-out fold of the current cross-validation split. FGSM and PGD are run across a wide range of attack strengths (in line with the robustness curve proposal of [Göpfert et al.(2020a)Göpfert, Göpfert, and Hammer]), namely ϵ∈[0,0.4]\epsilon\in[0, 0.4] in steps of 0.01 for MNIST, and ϵ=k/510\epsilon= k/510 for k=0,…,34k = 0, \ldots, 34, i.e. ϵ∈[0,0.067]\epsilon\in[0, 0.067], for CIFAR-10 and Imagenette-160. Regarding the attack configurations, FGSM and PGD are both ℓ∞\ell_\infty attacks. PGD uses TT steps with step size α=2.5ϵ/T\alpha= 2.5 \epsilon/T and a random start inside the ϵ\epsilon-ball, while the C&W attack uses a confidence of κ=0\kappa= 0, NN binary-search steps over the trade-off constant, and MM optimization steps per search step. The values of TT, NN, MM, and the number of random restarts are listed in Table 3. The C&W attack is unbounded and is reported as a mean ℓ2\ell_2 distortion rather than as an accuracy curve.

Experimental Results

We evaluate our approach on MNIST, CIFAR-10, and Imagenette-160 discussed in § 4.1. For each dataset, we compare six models: three heads (Linear, SVM, GLVQ) crossed with two embedding networks (standard and Lipschitz-constrained) (see § 3). Our goal is to separate the effect of the Lipschitz loss from the effect of the classification head. We report clean accuracy and adversarial accuracy under FGSM, PGD, and C&W attacks. Unless stated otherwise, all results use L=0.032L = 0.032 and Lipschitz weight λ=10\lambda= 10 (§ 4.4). All curves reported in Figure 3a – Figure 5c are the means over 5-fold cross-validation, with error bars representing the standard deviation across folds.

Figure 3. MNIST robustness. (3a,3b) Accuracy versus the ℓ∞\ell_\infty attack budget ϵ\epsilon. (3c) Mean ℓ2\ell_2 distortion required to flip the label; higher is more robust.

Robustness plots for FGSM, PGD, and C&W attacks on Imagenette-160

Figure 5. Imagenette-160 robustness. (5a,5b) Accuracy versus the ℓ∞\ell_\infty attack budget ϵ\epsilon. (5c) Mean ℓ2\ell_2 distortion required to flip the label; higher values indicate greater robustness.

The effect of the Lipschitz loss on clean accuracy depends strongly on the dataset (Table 4). On MNIST, the two Lipschitz models slightly exceed their unconstrained counterparts, while on CIFAR-10 the standard models lead. The Lipschitz loss gains 0.12 percentage points on MNIST and costs 2.66 on CIFAR-10, consistent with the known robustness–accuracy trade-off [46 ) Zhang, Yu, Jiao, Xing, El Ghaoui, and Jordan] () Tsipras, Santurkar, Engstrom, Turner, and Madry, Zhang et al. 2019). On Imagenette-160, the cost is largest, 8.01 points for the Linear models (65.19% vs. 73.20%), reflecting the slower convergence of the constrained embedding network under the matched 50-epoch budget, as shown by the loss curves in § 5.2.

ModelMNISTCIFAR-10Imagenette-160
Linear (standard)98.24 ±\pm 0.1078.97 ±\pm 0.4973.20 ±\pm 2.34
SVM (standard)98.34 ±\pm 0.1378.79 ±\pm 0.6078.14 ±\pm 1.06
GLVQ (standard)97.82 ±\pm 0.1679.18 ±\pm 0.5376.40 ±\pm 1.15
Lipschitz-Linear98.36 ±\pm 0.1076.31 ±\pm 0.9965.19 ±\pm 2.24
Lipschitz-SVM98.46 ±\pm 0.1076.82 ±\pm 0.9368.35 ±\pm 1.12
Lipschitz-GLVQ98.09 ±\pm 0.1776.49 ±\pm 1.2067.19 ±\pm 1.16

Table 4. Clean accuracy (%). Values are the mean ±\pm standard deviation over 5 folds.

The clean accuracy (without any adversarial attack at ϵ=0\epsilon= 0) of the GLVQ models are close to that of Linear and SVM on every dataset. GLVQ is 0.42 percentage points below Linear on MNIST and 1.74 points below SVM on Imagenette-160, while on CIFAR-10 the three heads are within 0.4 points of each other. Voráček and Hein [Voráček and Hein [49] similarly report 97.3% clean accuracy for ℓ2\ell_2-PNPC on MNIST. Zhang et al. [Zhang et al. (2019) Zhang, Yu, Jiao, Xing, El Ghaoui, and Jordan] observed a comparable trade-off on CIFAR-10, where TRADES (1/λ=61/\lambda= 6) achieved 84.92% clean accuracy against 56.61% robust accuracy under PGD20. Our clean accuracy is well below this, as expected given our relatively small embedding network with only 64 embedding dimensions and 3 layers; we cite the comparison to situate the size of the trade-off, not to claim equality.

Adversarial Robustness

At ϵ=0\epsilon= 0 (clean accuracy) all models achieve high accuracy. As ϵ\epsilon increases, performance degrades rapidly for the standard models. Lipschitz-GLVQ achieves the highest robustness on MNIST and CIFAR-10 (Figure 3a–Figure 4c). On Imagenette-160 the picture differs: under PGD, Lipschitz-SVM is the strongest model over the range where accuracies remain substantial, with Lipschitz-GLVQ second, while under FGSM attacks, Lipschitz-GLVQ leads from ϵ≈0.016\epsilon\approx0.016 onwards by a wide margin (10.8% against 3.7% at the largest budget). Among the unconstrained models, GLVQ is the most robust under FGSM attacks on all three datasets. This is likely because our embedding network already exhibits some Lipschitz continuity even without explicit training, while GLVQ’s margin-based decision rule provides a natural robustness certificate in the embedding space [Hammer et al. (2005) Hammer, Strickert, and Villmann, Saralajew et al. (2020b) Saralajew, Holdijk, and Villmann].

Figure 4. CIFAR-10 robustness. (4a,4b) Accuracy versus the ℓ∞\ell_\infty attack budget ϵ\epsilon. (4c) Mean ℓ2\ell_2 distortion required to flip the label.

A key observation is that the improvement of Lipschitz-Linear and Lipschitz-SVM over their standard counterparts is strongest on MNIST. On CIFAR-10 (Figure 4a) and Imagenette-160 (Figure 5a), both also clearly outperform their unconstrained counterparts at small attack budgets. This advantage diminishes as ϵ\epsilon increases, and on Imagenette-160 and under FGSM attacks on CIFAR-10, unconstrained GLVQ overtakes them at higher attack strengths.

The results on MNIST are the clearest: under FGSM attacks (Figure 3a), Lipschitz-GLVQ retains ≈81%\approx81\% accuracy at ϵ=0.20\epsilon= 0.20 and ≈43%\approx43\% at ϵ=0.40\epsilon= 0.40. Under the stronger PGD adversary (Figure 3b), it retains ≈53%\approx53\% at ϵ=0.20\epsilon= 0.20, while every unconstrained model has collapsed to near-zero by ϵ≈0.19−0.20\epsilon\approx0.19 - 0.20. Degradation occurs only when the images are altered so strongly that a human observer might not recognize them, which is consistent with the goal of interpretability: a model should change its prediction only when the input has changed enough to change a human’s judgment [47 )Vadillo, Santana, and Lozano]. On CIFAR-10 and Imagenette-160, however, a human observer would likely not change their predicted class even at the largest ϵ\epsilon tested, so the current level of robustness is still insufficient for true interpretability, despite the significant improvement over the baseline.

The C&W results (Figure 3c, Figure 4c, Figure 5c) address the same question from a minimum-distortion perspective. On MNIST, the mean ℓ2\ell_2 distortion needed to flip a label rises from 1.57 to 2.93 for the GLVQ pair (1.87×\times) and from 0.91 to 1.79 for both Linear and SVM. On CIFAR-10 the two GLVQ variants are indistinguishable (0.194 against 0.194), so Lipschitz training brings no measurable C&W advantage there. On Imagenette-160, the ordering is favorable but within one standard deviation (0.387 →\to 0.414). On CIFAR-10 and Imagenette-160, both prototype models require clearly more distortion than the four linear heads; on MNIST this holds for Lipschitz-GLVQ (2.93), while standard GLVQ (1.57) sits below the two Lipschitz linear heads.

In comparison to prior work, Voráček and Hein [Voráček and Hein [49] report that ℓ2\ell_2-PNPC attains 96.2% certified accuracy on MNIST at ϵ1=1\epsilon_1 = 1, outperforming ℓ2\ell_2-GLVQ (94.2%) and 1-NN (95.0%). Our empirical results align with these findings, though our model is not certified in the same sense, because we enforce local Lipschitz continuity of the embedding only stochastically via training rather than proving a strict, global bound on the whole model. Saralajew et al. [Saralajew et al. (2020a)Saralajew, Holdijk, Rees, and Villmann] found that GLVQ and GTLVQ exhibit high base robustness without adversarial training, and that adversarial examples against them resemble semantically meaningful interpolations toward the target class rather than arbitrary noise; our observation that standard GLVQ beats Linear and SVM in terms of robustness supports this.

These results give partial support to our hypothesis. Lipschitz training on its own is clearly not sufficient: with a softmax head it buys almost nothing beyond MNIST, which is consistent with the observation that the most Lipschitz-stable model is the useless constant function [59]. The broader picture on the margin-based heads is more mixed than the theory would suggest. When comparing Lipschitz training to the choice of classifier, GLVQ appears to yield a clearer effect: unconstrained GLVQ already beats Linear and SVM everywhere, the SVM benefits much less from Lipschitz training compared to GLVQ, and on Imagenette-160 Lipschitz-GLVQ does not beat regular GLVQ. We return to these two discrepancies in § 6.

Hyperparameter Sensitivity Analysis

The Lipschitz loss depends on three free parameters: the desired Lipschitz constant LL, the weight λ\lambda, and the neighborhood standard deviation σ\sigma. Because these interact in a way that is not obvious from the loss definition alone, we visualize their effect in Figure 6, Figure 7 and Figure 8.

Figure 8. Effect of the neighborhood standard deviation σ∈{0.03,0.07,0.1,0.2,0.3,0.5}\sigma\in\{0.03, 0.07, 0.1, 0.2, 0.3, 0.5\} at fixed L=0.032L = 0.032 and λ=10\lambda= 10, over 50 epochs. Solid curves: weighted Lipschitz loss λ⋅LLipschitz\lambda\cdot\mathcal{L}_{\mathrm{Lipschitz}} (left, log axis). Dashed: classification loss (right axis). Mean over 5 folds; error bars give one standard deviation across folds.

Lipschitz weight λ\lambda. On MNIST (Figure 6a) the weighted Lipschitz loss rises over the first 5–10 epochs, peaks, and then decays. This is the signature of the two objectives competing. At initialization, the embedding network is near-random, the embedding is compressed, and the

constraint is barely violated; as the classification loss falls, the embedding network expands the embedding to separate classes, increasing its local sensitivity. The Lipschitz loss only begins to dominate once this expansion has occurred. The curves have the same shape for all λ\lambda values; only their magnitude changes. Thus, λ\lambda controls the strength of the Lipschitz loss but does not change how the loss behaves over time. The classification-loss curves for all six λ\lambda are indistinguishable and converge to near zero by epoch 30, meaning that, for MNIST, λ\lambda can be increased without a strong impact on the final training loss. Accordingly, the test accuracy also hardly changes across different λ\lambda values (Table 4).

Lipschitz constant LL: Figure 7 shows a monotone ordering across all three datasets: the smaller the Lipschitz constant LL, the larger the residual Lipschitz loss at convergence.

For L≳0.707L \gtrsim0.707, the constraint has almost no effect, indicating that the regular training of ϕ\phi already ensures that distances are never enlarged beyond this factor. For example, on MNIST (Figure 7a), when LL is large (0.707, 0.775, or 1), the Lipschitz loss drops below 10−410^{-4} and often drops to zero and becomes unstable, with error bars varying across multiple orders of magnitude. Correspondingly, the constraint is violated on 25% of sampled pairs on MNIST and 95% on CIFAR-10 at our operating point (Table 5). In practice, the Lipschitz loss effectively vanishes, so it contributes almost no gradient and we would not expect the learned embedding to differ from the unconstrained one. We did not test this directly at these values of LL.

At the other extreme, the L=0L = 0 and L=0.032L = 0.032 curves are nearly identical on all three datasets. This is worth noting because L=0L = 0 punishes any distance between ϕ(x)\phi(x) and ϕ(y)\phi(y), boiling down to something akin to L2 regularization on ϕ\phi. Still, it appears that the constraint degrades gracefully toward L=0L = 0, indicating that the models do not collapse. Clean accuracy stays high, and on MNIST the classification loss is unaffected. Likely, this stability can be effect that the classification loss and the Lipschitz loss act as opposing forces: collapsing all distances would lead to high classification loss, such that a stable tradeoff occurs. Where the tradeoff lies is, in turn, regulated by λ\lambda.

Two considerations govern the choice of LL. Theorem 1 guarantees local ϵ/L\epsilon/L-robustness, so we want LL to be as small as possible to achieve the highest possible robustness; at the same time, a smaller LL constrains the embedding network more strongly and, thus, may reduce accuracy. Figure 7 shows how the Lipschitz loss develops for different LL. For L≳0.707L \gtrsim0.707, the Lipschitz loss collapses below 10−410^{-4} and becomes erratic, so the constraint is essentially never violated and contributes almost no gradient. At the other end, the L=0L = 0 and L=0.032L = 0.032 curves nearly coincide, indicating that the Lipschitz loss saturates below 0.032 and reducing LL further changes nothing. Within the active range, the dashed classification-loss curves are largely unaffected by LL, and clean accuracy stays within one percentage point of the baseline (Table 4). We therefore adopt L=0.032L = 0.032: the smallest value at which the Lipschitz loss is still distinguishable from the degenerate L=0L = 0 case and at which accuracy is still little affected.

Note that LL is not the actual Lipschitz constant of the trained embedding network. LL would only be the Lipschitz constant if the Lipschitz loss would reach zero for infinitely long training (i.e.: infinitely many stochastic samples across the neighborhood). Figure 7 shows a non-zero residual for every L≤0.224L \le0.224. The constraint is violated on 25% of sampled pairs on MNIST and 95% on CIFAR-10 (Table 5). To quantify how far the trained network is from the requested bound, we report the empirical stretch ratio ∥ϕ(x)−ϕ(y)∥/∥x−y∥\lVert\phi(x) - \phi(y)\rVert/\lVert x - y\rVert, averaged over exactly those Gaussian neighbor pairs (x,y)(x,y) on which the Lipschitz loss is evaluated. We emphasize that this is not the Lipschitz constant of ϕ\phi, which is NP-hard to compute even for two-layer networks [48]. It is a sample average over a finite set of directions and therefore a lower bound on the true constant (the maximum over all pairs can only be larger). Its mean value is 0.028 on MNIST and 0.081 on CIFAR-10, against L=0.032L = 0.032 – close on MNIST, exceeded by a factor of 2.5 on CIFAR-10. Consequently, ϵ/L\epsilon/L is not a certified radius, and the robustness reported in § 5 is empirical throughout. This matters because L→0L \to0 is the degenerate limit where all points are mapped to the same constant embedding [59].

Table 5 shows that the violation rate (measured on randomly sampled Gaussian neighbors of the training points, using the same pairs the Lipschitz loss is computed on) falls monotonically with λ\lambda on all three datasets. This confirms that λ\lambda controls how strictly the constraint is enforced, and suggests that the violation rate is never driven to zero, so target LL is not attained everywhere in either setting. The cost differs sharply by dataset: raising λ\lambda from 1 to 100 reduces clean accuracy by 0.4 points on MNIST, 11 on CIFAR-10, and 46 on Imagenette-160, which is why we adopt λ=10\lambda= 10 rather than a larger value.

Neighborhood standard deviation σ\sigma. The third parameter controls where the constraint is sampled rather than how tightly it is enforced. If σ\sigma is increased, neighbors yiy_i are drawn further away from the training point xix_i, so the Lipschitz condition of Eq. (4) is checked over a wider region. Figure 8 shows a monotone ordering on CIFAR-10 and Imagenette-160. The results indicate that the larger σ\sigma, the smaller the residual Lipschitz loss at convergence, spanning about one order of magnitude between σ=0.03\sigma= 0.03 and σ=0.5\sigma= 0.5. This is a property of the loss rather than of the embedding network, because the term L2d(xi,yi)2L^2d(x_i,y_i)^2 grows quadratically with the distance to the neighbor while the embedding displacement saturates. Consequently, the constraint is satisfied more often for distant neighbors, and the Lipschitz loss in Equation (4) is inactive on those neighbor pairs. A small residual at large σ\sigma therefore indicates that the constraint is loosely enforced. On MNIST, only σ=0.03\sigma= 0.03 is separated from the rest, and the remaining values coincide, indicating that the Lipschitz loss is insensitive to σ\sigma above 0.1 on this dataset. The classification loss behaves in the opposite direction on CIFAR-10 and Imagenette-160. On both of these datasets, the smallest σ\sigma converges most slowly, and the ordering is preserved throughout training, while on MNIST the curves are indistinguishable. We therefore select σ=0.1\sigma= 0.1 for MNIST and σ=0.03\sigma= 0.03 for CIFAR-10 and Imagenette-160, which are the smallest values at which the Lipschitz loss is still clearly active.

Discussion

The results indicate that adversarial robustness is not the consequence of a single design choice or hyperparameter. Several factors act together, and in our study the classification head emerges as the dominant one, rather than the Lipschitz loss alone. Under FGSM and PGD attacks at moderate and large budgets, the two prototype-based models (GLVQ and Lipschitz GLVQ) are more robust than the linear and SVM classifiers trained on the same frozen embedding. Standard GLVQ, without any Lipschitz loss, already outperforms both Linear and SVM under FGSM, PGD, and C&W attacks. This reproduces the observation by Saralajew et al. [Saralajew et al.(2020a)Saralajew, Holdijk, Rees, and Villmann] that GLVQ models possess a high base robustness before any explicit certification, and it is consistent with the margin-based view of these models: the decision rule of a nearest prototype classifier carries its own hypothesis margin, whereas linear+softmax classification carries none [8 )Hammer, Strickert, and Villmann] ()Crammer, Gilad-bachrach, Navot, and Tishby, Hammer et al. 2005). This gain is obtained at a cost to clean accuracy, which remains below two percentage points on all three benchmarks (Table 4), and without giving up interpretability, since the prototypes are stored in X\mathcal{X} and are initialized from training images. The Lipschitz constraint plays a complementary role: it is what makes the guarantee of Theorem 1 available at all. Without a controlled value of LL, that is, without a bound on how much the embedding network can stretch distances in the input space, robustness guarantees in the embedding space tell us nothing about robustness in the original input space, because an unconstrained embedding network can map an arbitrarily small input perturbation to an arbitrarily large attack in the embedding space, thus exceeding the robustness guarantees. The measured effect of the Lipschitz loss on robust accuracy, however, depends strongly on the choice of classification head, even though the embedding itself is identical for all three. The improvement of Lipschitz-Linear and Lipschitz-SVM over their unconstrained counterparts is confined to MNIST. On CIFAR-10, Lipschitz-Linear and Lipschitz-SVM fail to outperform their non-Lipschitz counterparts, and on Imagenette-160 the constrained variants are marginally worse against stronger attacks. The two margin-based heads on top of the embedding model therefore do not benefit equally from the Lipschitz loss. Figure 7 also visualizes the range in which the Lipschitz loss has any effect. For L≥0.707L \ge0.707, the loss falls below 10−410^{-4}, indicating that the embedding network that does not stretch distances beyond that factor. So setting LL this large does not have any effect; the regular training of the embedding network would have ensured this Lipschitz constant anyway. By contrast, for L=0.032L = 0.032, the Lipschitz loss remains nonzero throughout training on all three datasets. The difference between Lipschitz-GLVQ and standard GLVQ on MNIST and CIFAR-10 can thus be attributed to the effect of the Lipschitz loss. More broadly, Figure 7 indicates that the Lipschitz loss has no effect for L≥0.707L \ge0.707 and saturates for L≤0.032L \le0.032. Hence, we select L=0.032L = 0.032. For this setting, the clean (i.e. not attacked) accuracy remains almost unaffected while achieving as strong robustness as possible. For example, on MNIST, where the input dimension is low and the embedding space is only 16-dimensional, Lipschitz-GLVQ retains roughly 52.9% accuracy under PGD attacks at ϵ=0.20\epsilon= 0.20 (§ 5), an attack strength at which every unconstrained model has collapsed to zero.

Computationally, Lipschitz training is relatively cheap: it costs one additional forward pass through ϕ\phi per mini-batch, against the several forward–backward passes per sample required by the inner maximization of adversarial training [31 )Madry, Makelov, Schmidt, Tsipras, and Vladu]. On Imagenette-160, this cheapness is partly offset by slower convergence, as the constrained embedding network reaches only 65.1% validation accuracy at 50 epochs, compared to 74.7% for the baseline.

As discussed in § 4.4, the Lipschitz constant LL, the Lipschitz weight λ\lambda, and the neighborhood width σ\sigma do not act independently, making hyperparameter optimization a multi-dimensional (and multi-objective; due to the accuracy-robustness tradeoff) optimization problem. Still, it is possible to characterize the different roles of LL, λ\lambda, and σ\sigma, and provide some guidance for reasonable settings. LL limits how much the embedding network may stretch a small input perturbation into a displacement in the embedding space, λ\lambda determines how aggressively that limit is enforced during training, and σ\sigma decides whether the constraint is checked only very close to the training data or over a broader region. A larger λ\lambda enforces the constraint more strictly and drives the violation rate down, while a larger LL relaxes it, so the two can effectively cancel. If σ\sigma is too small, the embedding network may satisfy the Lipschitz bound near the training points while remaining unconstrained further away, leaving gaps that an attack can exploit. A small LL therefore does not guarantee robustness if λ\lambda is too weak to enforce it or σ\sigma is too narrow to cover the relevant region, just as a large λ\lambda with a wide σ\sigma is of little use if LL is set so loosely that the constraint is vacuous. Only when all three are chosen in concert does the constraint provide a transferable statement about robustness. The three parameters are not tunable independently. Reporting only one without the other two is insufficient for reproducibility. Two additional observations follow. First, the architecture of the embedding network may be just as important as the Lipschitz loss. We used a small embedding network for fair comparison, not for peak performance. A larger embedding network may raise both clean and robust accuracy. On MNIST, clean accuracy is high (98%), so the model handles a wide range of attacks. On CIFAR-10 and Imagenette-160, clean accuracy is lower (around 70%). There is less room to spare, so performance drops faster. Second, the interaction differs by dataset, so a single global setting of (L,λ,σ)(L, \lambda, \sigma) is unlikely to be appropriate everywhere.

The behavior across datasets and attacks is not uniform. Lipschitz-GLVQ is the strongest model under FGSM attacks on all three datasets once the attack limit exceeds a small threshold. Under PGD attacks, the six models are close on MNIST at small budgets but separate sharply at larger ones, while on CIFAR-10 Lipschitz-GLVQ retains a clear advantage over the linear and SVM heads. On Imagenette-160, however, Lipschitz-SVM is the strongest model under PGD attacks over the range where accuracies remain substantial, with Lipschitz-GLVQ second. Since both are maximum-margin heads, which Theorem 1 treats alike, we take this as another instance of the SVM/GLVQ difference that needs to be explored further. The C&W results are the least consistent: on MNIST the ordering is clear, with the mean ℓ2\ell_2 distortion of the GLVQ pair rising from 1.57 to 2.93, and on Imagenette-160 it is favorable (0.387 →\to 0.414), whereas on CIFAR-10 the two are indistinguishable (0.194 against 0.194). The C&W is also the least stable attack across runs. Between two runs on Imagenette-160, the unconstrained baselines shifted as well (Linear 0.36 →\to 0.46, SVM 0.32 →\to 0.40), although they contain no Lipschitz loss and are unaffected by LL and λ\lambda. Part of the variation therefore originates in the internal routines of the C&W attacks, rather than in the models (§ 5.2). Beyond this, σ\sigma is expressed in fixed input units and so covers a smaller relative region of a 64×64×364 \times64 \times3 image than of a 28×2828 \times28 digit. Resolution-aware choice of σ\sigma would be a natural next step. Imagenette-160 also has by far the smallest training set (9,469 images), and the matched 50-epoch budget penalizes the constrained embedding network on this dataset given its slower convergence. The FGSM results on this dataset, which are stable across folds, show Lipschitz-GLVQ ahead of both linear heads, and we take those as the primary evidence.

Overall, our key results are: The combination of Lipschitz training for a deep embedding network and a GLVQ classifier on the embeddings yields a promising deep prototype classification approach. Our theory suggests that this is due to a transfer of robustness guarantees in the embedding space to the overall model, thanks to Lipschitz continuity. However, these results are complicated by the fact that Lipschitz-SVM falls short of Lipschitz-GLVQ (even though we would have expected similar results) and that GLVQ performs very similar to Lipschitz-GLVQ for some dataset-attack-combinations.

Limitations

While we hope to have substantially pushed forward the research on deep embedding networks combined with robust, shallow classifiers, we also acknowledge multiple limitations:

  • Local robustness is assumed by our theory, not proven. We rely on prior robustness proofs. However, the fact that Lipschitz SVM does not perform as well as Lipschitz-GLVQ indicates that this may be problematic. A deeper theoretical analysis of the difference between SVM and GLVQ robustness guarantees is required.

  • Lipschitz training does not guarantee a local Lipschitz bound but only punishes violations for stochastically sampled neighbors. This leaves room for future work on more effective training schemes (without losing the computational efficiency of our approach).

  • While empirical improvements are striking against FGSM attacks, robustness gains against stronger attacks (PGD and C&W) are more modest. Accordingly, stronger guarantees may be needed.

  • Our evaluation is limited to three attacks, each using a single random restart. A more comprehensive evaluation using an ensemble-based approach such as AutoAttack [9] would provide a stronger test of the observed robustness and is left for future work.

  • Our experiments are limited to three data sets and three classifiers. Further classifiers, especially those with robustness guarantees, could be investigated in subsequent work. Similarly, further datasets, involving more modalities than just image data, could be investigated.

Conclusion

Broadly speaking, our motivation stems from the idea that robustness and interpretability are connected by a similar underlying question: does a model only change its prediction when the input has accumulated enough evidence to support a different label? The MNIST results provide some support for this interpretation. In this setting, Lipschitz-GLVQ changes its prediction only at perturbation levels where the digit itself becomes difficult for a human to recognize. This correspondence is less clear for CIFAR-10 and Imagenette-160. For these datasets, a human observer would generally still assign the original label within the tested perturbation budgets, even though the models may already have changed their predictions. Therefore, while Lipschitz-GLVQ improves robustness on the color datasets, the improvement is not yet strong enough to fully support the proposed interpretability argument, and the intersection of robustness and interpretability remains a promising direction for future work.

References

  1. [1][Angelov and Soares(2020)] Plamen Angelov and Eduardo Soares. Towards deep machine reasoning: a prototype-based deep neural network with decision tree inference. pages 2092–2099, 2020. doi:10.1109/SMC42975.2020.9282812.DOI
  2. [2][Biehl et al.(2016)Biehl, Hammer, and Villmann] Michael Biehl, Barbara Hammer, and Thomas Villmann. Prototype-based models in machine learning. Wiley Interdisciplinary Reviews: Cognitive Science, 7(2):92–111, 2016. doi:10.1002/wcs.1378.DOI
  3. [3][Biran and Cotton(2017)] Or Biran and Courtenay Cotton. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), volume 8, pages 8–13, 2017. URL https://api.semanticscholar.org/CorpusID:3911355.
  4. [4][Carlini and Wagner(2017)] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017. doi:10.1109/SP.2017.49.DOI
  5. [5][Carlini et al.(2019)Carlini, Athalye, Papernot, Brendel, Rauber, Tsipras, Goodfellow, Madry, and Kurakin] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019. doi:10.48550/arXiv.1902.06705.DOI
  6. [7][Chen et al.(2019)Chen, Li, Tao, Barnett, Rudin, and Su] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. volume 32, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/adf7ee2dcf142b0e11888e72b43fcb75-Paper.pdf.
  7. [8][Crammer et al.(2002)Crammer, Gilad-bachrach, Navot, and Tishby] Koby Crammer, Ran Gilad-bachrach, Amir Navot, and Naftali Tishby. Margin analysis of the lvq algorithm. In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems, volume 15, pages 479–486. MIT Press, 2002. URL https://proceedings.neurips.cc/paper_files/paper/2002/file/bbaa9d6a1445eac881750bea6053f564-Paper.pdf.
  8. [9][Croce and Hein(2020)] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 2206–2216. PMLR, 2020. URL https://proceedings.mlr.press/v119/croce20b.html.arxiv.org/abs/2003.01690
  9. [15][Finlay and Oberman (2021)] Chris Finlay and Adam M Oberman. Scaleable input gradient regularization for adversarial robustness. Machine Learning with Applications, 3:100017, 2021. URL arXiv:2503.20454.
  10. [20][Hammer and Villmann (2002)] Barbara Hammer and Thomas Villmann. Generalized relevance learning vector quantization. Neural Networks, 15(8-9):1059–1068, 2002. doi:10.1016/S0893-6080(02)00079-5.DOI
  11. [22][Hoffman et al.(2019)Hoffman, Roberts, and Yaida] Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. 2019.arxiv.org/abs/1908.02729
  12. [23][Howard(2019)] Jeremy Howard. Imagenette: A smaller subset of 10 easily classified classes from imagenet, March 2019. URL https://github.com/fastai/imagenette.
  13. [26][Khalid and Paaßen(2026)] Adia Khalid and Benjamin Paaßen. Adversarial robustness by combining prototype models with lipschitz training. In Michel Verleysen, editor, 34th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), 2026. in press.
  14. [27][Kohonen(1990)] Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9): 1464–1480, 1990. doi:10.1109/5.58325.DOI
  15. [28][Krizhevsky(2009)] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. URL https://api.semanticscholar.org/CorpusID:18268744.
  16. [29][LeCun et al.(2010)LeCun, Cortes, and Burges] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
  17. [30][Li et al.(2018)Li, Liu, Chen, and Rudin] Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. 32(1), 2018.
  18. [31][Madry et al.(2017)Madry, Makelov, Schmidt, Tsipras, and Vladu] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv:1706.06083, 2017. doi:https://doi.org/10.1609/aaai.v32i1.11771.
  19. [32][Matyasko and Chau(2017)] Alexander Matyasko and Lap-Pui Chau. Margin maximization for robust classification using deep learning. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 300–307. IEEE, 2017. doi:10.1109/IJCNN.2017.7965869.
  20. [33][Moosavi-Dezfooli et al.(2016)Moosavi-Dezfooli, Fawzi, and Frossard] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2574–2582, 2016.
  21. [36][Ross and Doshi-Velez (2018)] Andrew Ross and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. doi:https://doi.org/10.1609/aaai.v32i1.11504.
  22. [40][Sato and Yamada(1995)] Atsushi Sato and Keiji Yamada. Generalized learning vector quantization. Advances in neural information processing systems, 8, 1995. URL https://proceedings.neurips.cc/paper_files/paper/1995/file/9c3b1830513cc3b8fc4b76635d32e692-Paper.pdf.
  23. [42][Seo and Obermayer(2003)] Sambu Seo and Klaus Obermayer. Probabilistic learning vector quantization. Neurocomputing, 55(1-2):189–211, 2003. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/d54e99a6c03704e95e6965532dec148b-Paper.pdf.pubmed.ncbi.nlm.nih.gov/12816567
  24. [44][Snell et al.(2017)Snell, Swersky, and Zemel] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/cb8da6767461f2812ae4290eac7cbc42-Paper.pdf.arxiv.org/abs/1703.05175
  25. [45][Szegedy et al.(2013)Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.arxiv.org/abs/1312.6199
  26. [46][Tsipras et al.(2018)Tsipras, Santurkar, Engstrom, Turner, and Madry] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. 2018.arxiv.org/abs/1805.12152
  27. [47][Vadillo et al.(2025)Vadillo, Santana, and Lozano] Jon Vadillo, Roberto Santana, and Jose A Lozano. Adversarial attacks in explainable machine learning: A survey of threats against models and humans. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 15(1):e1567, 2025. doi:https://doi.org/10.1002/widm.1567.
  28. [48][Virmaux and Scaman(2018)] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in neural information processing systems, 31, 2018.
  29. [49][Voráček and Hein(2022)] Václav Voráček and Matthias Hein. Provably adversarially robust nearest prototype classifiers. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022. URL https://proceedings.mlr.press/v162/voracek22a.html.arxiv.org/abs/2207.07208
  30. [50][Wang et al.(2018)Wang, Jha, and Chaudhuri] Yizhen Wang, Somesh Jha, and Kamalika Chaudhuri. Analyzing the robustness of nearest neighbors to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 5133–5142. PMLR, 2018.arxiv.org/abs/1706.03922
  31. [52][Xu et al.(2020)Xu, Ma, Liu, Deb, Liu, Tang, and Jain] Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang, and Anil K Jain. Adversarial attacks and defenses in images, graphs and text: A review. International journal of automation and computing, 17(2):151–178, 2020. doi:https://doi.org/10.1007/978-3-030-32236-6_51.
  32. [53][Yang et al.(2020)Yang, Rashtchian, Wang, and Chaudhuri] Yao-Yuan Yang, Cyrus Rashtchian, Yizhen Wang, and Kamalika Chaudhuri. Robustness for non-parametric classification: A generic attack and defense. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), pages 941–951. PMLR, 2020.arxiv.org/abs/1906.03310
  33. [59][Zühlke and Kudenko (2025)] Monty-Maximilian Zühlke and Daniel Kudenko. Adversarial robustness of neural networks from the perspective of lipschitz calculus: A survey. ACM Computing Surveys, 57(6):1–41, 2025. doi:10.1145/3648351.DOI

Paper details

vlnc:W_nhwzv485
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents