Hyperspherical Simplex Representations for Backward-Compatible Retrieval

Website Code results on standard compatibility benchmarks, including scenarios with expanded class sets, more expressive architectures, and independently trained models from public repositories.

Introduction

Training modern AI models has become increasingly expensive, limiting accessibility to a few well-resourced organizations [76, 27, 47, 3, 110, 107, 94]. Despite reductions in model parameters and computing costs [27], training from scratch, fine-tuning, and inference remain economically challenging [100, 67]. As a result, models are increasingly released through public repositories [59, 104, 103] or provided as services through APIs, simplifying application development and enabling widespread adoption across domains.

As these models become more integrated into real-world systems, the need to update them is also growing. This demand is driven by several factors [108] (Raffel 2023), including evolving training strategies [12, 11, 29, 88], advancements in model architectures [96, 49], the availability of higher-quality datasets [37], the expansion of training classes, or extended training periods [8] (Raffel 2023). Such updates not only improve performance but also bring the rapid progress of the field into deployed models, simplifying their adoption despite the continued growth in the scale and complexity of models, datasets, and infrastructures [13, 92].

However, a significant drawback of model updates is that they can compromise compatibility with earlier versions, altering the behavior of applications or services built upon them (Raffel, 2023). For example, image retrieval services might experience unexpected changes in image rankings following a model update, forcing users to adapt when previously top-ranked images no longer appear first [88]. Similarly, end users, such as drivers of semi-autonomous cars, often develop a mental model of the machine learning system’s capabilities [6]. When updates modify system behavior, users must adjust their mental model of the system’s functionality, a task that can be challenging, dissatisfying, and even unsafe [7, 95]. Recent observations have highlighted similar issues with large language models (LLMs) [29, 35], raising concerns about AI safety and alignment. This is particularly critical when decisions and textual outputs from these agents are translated into actions [2, 66]. In such settings, it is essential that the predictions of the updated model mimic those of the old model for any correct classification [63, 109, 84].

Updating a model while ensuring compatibility with its earlier versions has been explored along three complementary dimensions: 1) Weights-level, which explores the merging of model parameters under the assumption of linear mode connectivity, to unify multiple versions into a single, robust model [34, 105, 1]; 2) Representation-level, which seeks to learn backward-compatible representations that are comparable across model updates, so they can be used interchangeably over time [88, 62, 12]; and 3) Classifier decision-level, which ensures that the updated model preserves the correct predictions of the old model, preventing negative flips [63, 109, 116]. They target distinct notions of compatibility and accordingly differ in their objectives, mechanisms, and evaluation criteria.

Among these paradigms, we focus on representation-level compatibility, which is particularly relevant to retrieval systems, where feature representations produced by different model versions must remain directly comparable. Backward-compatibility training, introduced by Shen et al. [88] to avoid image re-indexing of the gallery set, has been established as a standard training paradigm to achieve backward-compatibility between model representations. However, most existing studies on backward-compatibility require training a new model using the old model as a reference [88, 62, 28, 69, 119]. This approach has become increasingly challenging due to restricted access to old model architectures and weights—often limited by proprietary constraints and non-disclosure policies—as well as the high computational costs associated with training large-scale modern AI models. Recent studies [9, 11, 12] have addressed this limitation by learning compatible representations without explicitly using the previous model as a reference. However, these approaches require all parties to agree in advance on a shared, pre-allocated fixed classifier [72, 73, 121].

In this paper, we show that hyperspherical simplex representations derived from the softmax outputs or logits of independently trained³ models provide backward-compatible retrieval. Our formulation leverages the simplex geometry induced during training by the softmax function and one-hot encoded class labels as a model-agnostic reference, yielding hyperspherical simplex representations that are aligned across model updates. Since their simplex geometries evolve with the introduction of new classes, we propose a deterministic linear transformation that compensates for the resulting angular misalignment between the prototypes of existing classes and their earlier counterparts, thereby preserving their alignment. We theoretically analyze the formal backward-compatibility definition and identify geometric conditions under which its inequalities [88] hold in expectation, given explicit assumptions on the feature distributions and their concentration across updates. We then show that these conditions are satisfied by aligned regular-simplex representations, and therefore by the proposed hyperspherical simplex representations, which exhibit this geometry. Although the proposed hyperspherical simplex representations satisfy the geometric conditions established in our analysis, their dimensionality increases linearly with the number of training classes, resulting in substantial storage costs for large-scale galleries. To mitigate this limitation, we adopt a top-$k$ sparsification strategy motivated by the rank-preserving property of the softmax function and the top-$k$ calibration of the softmax cross-entropy loss [52, 111], which reduces gallery storage with negligible changes in retrieval accuracy and without degrading compatibility.

Our main contributions are the following:

1. We introduce a novel approach for backward-compatible retrieval based on hyperspherical simplex representations derived from softmax outputs and logits.

2. We provide a theoretical analysis of the formal definition of backward-compatibility, identifying geometric conditions under which its inequalities are satisfied in expectation, given explicit distributional and concentration assumptions. We show that hyperspherical simplex representations derived from softmax outputs and logits satisfy these geometric conditions.

3. We propose a top-$k$ sparsification method for hyperspherical simplex representations, motivated by the rank-preserving property of the softmax function and the top-$k$ calibration of the softmax cross-entropy loss, to reduce representation storage.

4. We achieve competitive results on standard compatibility benchmarks, obtaining substantial improvements in scenarios involving a large number of model updates, without using additional compatibility-specific losses, learned mappings, or modifications to the model architecture. We provide empirical evidence that representations derived from softmax outputs and logits substantially improve backward-compatibility, even between independently pretrained models obtained from a public repository.

The remainder of the paper is organized as follows. In § 2, we review related work on backward-compatibility. In § 3, we derive hyperspherical simplex representations from softmax outputs and logits and analyze their backward-compatibility under stated theoretical assumptions. In § 4, we present extensive experiments and compare our proposed method with state-of-the-art alternatives. Finally, in § 5, we draw conclusions and discuss the limitations of this work.

Related Work

Backward-compatibility was first formalized by Shen et al. [88], who defined both theoretical and empirical criteria to verify whether learned representations are interoperable across model updates, thus eliminating the need to re-index the gallery set in retrieval systems. To learn backward-compatible representations, they proposed Backward-Compatible Training (BCT), where the previously learned classifier is frozen and serves as a fixed reference during training of the new model, aligning new feature vectors with their corresponding old class prototypes. Subsequent work proposed additional regularization objectives to enforce alignment between old and new representations [16, 28, 62, 114, 115, 69, 106, 35]. In Meng et al. [62], a combination of loss functions is used to align class means between different model updates and to achieve more compact intra-class distributions while learning new information, directly optimizing a point-to-set formulation of backward-compatibility. In Zhang et al. [114], compatible representations are learned with a contrastive loss that encourages each new-to-old positive pair to be more similar than both new-to-old and new-to-new negative pairs. Pan et al. [69] introduce an adversarial discriminator within BCT to minimize the distribution gap between features from the old and new models. In these regularization-based approaches, due to the constraints imposed to achieve compatibility, the performance of the new backward-compatible model often does not reach that of a newly independently trained model [119], trained without using the old model as a reference. To address this issue, Zhou et al. [119] and Ricci et al. [81] focus on modifying the model architecture by expanding the feature space at each model update, such that the original subspace preserves compatibility with previous models, while information from the new data is learned in the additional dimensions. However, even if compatibility is achieved while retaining the discriminative performance of a newly independently trained model, these approaches require explicit modifications to the model architecture that introduce growing complexity with each successive feature space expansion.

Along a different line, several studies have proposed mapping-based approaches to achieve compatibility across model updates [21, 99, 62, 79, 44, 46, 82]. These methods learn mapping functions to align corresponding representations of the same inputs obtained from two different models, enabling direct comparison between them. Such approaches are closely related to prior work in cross-model and cross-modal representation learning, where alignment between feature spaces of different models or modalities is a central objective [65, 18, 19, 57, 58, 83].

However, feature-space alignment alone does not guarantee satisfaction of the formal compatibility inequalities, and mapping-based approaches introduce significant overhead in scenarios with many model updates, as they require composing multiple learned mapping functions across successive updates. To address these limitations, the solution proposed by Biondi et al. [9, 11, 12] involves learning with a pre-allocated $d$-simplex fixed classifier, which enforces feature alignment across updates by fixing the position of class prototypes a priori (i.e., from the beginning of training), while accommodating new classes within pre-allocated regions of the feature space. Although this approach achieves compatibility without using the old model as a reference, it relies on the same pre-allocated fixed classifier across model updates and requires architectural modifications.

In this paper, we show that hyperspherical representations with an aligned regular-simplex geometry can be obtained directly from the logits or softmax outputs of independently trained models, without requiring a pre-allocated fixed classifier shared across updates. Building on this observation, we derive hyperspherical simplex representations for backward-compatible retrieval that do not require additional compatibility-specific losses, learned mapping functions, or modifications to the model architecture. At the theoretical level, while Biondi et al. [12] numerically show that stationary $d$-simplex representations satisfy the compatibility inequalities in expectation, our analysis complements this result by providing closed-form geometric conditions for aligned hyperspherical representations under explicit distributional model assumptions.

Compatibility of Hyperspherical Simplex Representations

In this section, we first introduce hyperspherical simplex representations derived from softmax outputs and logits. We then identify geometric conditions under which the compatibility inequalities of Shen et al. [88] hold in expectation given explicit assumptions on the feature distributions and their increasing concentration across model updates. Finally, we show that the proposed representations satisfy these conditions.

Preliminaries and Motivation

According to Shen et al. [88], backward-compatibility across model updates is achieved when the representation produced by the updated model at step $t$ can be compared with the representation produced by the earlier model at step $k$. Formally, it requires that the following two inequalities hold:

where $\mathbf{h}^t$ and $\mathbf{h}^k$ are feature vectors extracted respectively with the updated model at step $t$ and the earlier model at step $k$, $d$ is a distance function (e.g., cosine distance), and $y_i$ and $y_j$ are class labels associated with $\mathbf{h}_i$ and $\mathbf{h}_j$, respectively.

In retrieval tasks, the inequalities in Eq. 1a and Eq. 1b suggest that the updated model representation, when used to compare against the old representation, should perform at least as well as the old one in grouping images from the same classes and separating those from different classes. While Eq. 1a and Eq. 1b describe desirable goals for compatible representations, directly verifying them requires exhaustive pairwise comparisons between the two representations, which is impractical at scale (Shen et al., [88]).

To obtain a numerical assessment of these inequalities, Biondi et al. [11, 12] estimate the corresponding expected distances through Monte Carlo integration under the unconstrained feature model (UFM) assumption (Mixon et al., [64]), considering feature distributions centered around the prototypes of a pre-allocated $d$-simplex fixed classifier. In their initial analysis, Biondi et al. [11] model class features as samples from Euclidean hyperballs and evaluate compatibility using Euclidean distance. Under this formulation, the different-class inequality (Eq. 1b) is not satisfied, and stationary $d$-simplex representations are therefore characterized as an optimal approximation to backward-compatibility. Extending this analysis, Biondi et al. [12] model class features as samples from hyperspherical caps on the unit hypersphere and evaluate compatibility using cosine distance. Assuming that the hyperspherical caps shrink across model updates, they demonstrate that both compatibility inequalities hold in expectation for representations learned with a pre-allocated $d$-simplex fixed classifier.

While these studies assess compatibility numerically for stationary $d$-simplex representations, we analytically characterize the compatibility inequalities and identify geometric conditions under which they are satisfied in expectation. Specifically, we model hyperspherical features using von Mises–Fisher (vMF) distributions on the unit hypersphere (Fisher, [33]; Banerjee et al., [5]), which yield closed-form expressions for the expected cosine distances in terms of class prototype directions and concentration parameters. Assuming that the concentration of the class distributions increases across model updates, we prove in Theorem 3.6 of § 3.4 that both inequalities of Def. 3.1 are satisfied in expectation if the following two geometric conditions hold:

• (a) corresponding class prototypes are aligned across updates;

• (b) the angle between prototypes of different classes is greater than $\pi/2$.

In particular, both conditions are satisfied by regular-simplex representations that are centered at the origin and aligned across model updates, as established in Cor. 3.7. Since softmax cross-entropy training with one-hot labels induces a simplex structure in both the softmax outputs and the logits, these provide a principled basis for constructing backward-compatible representations without compatibility-specific training. Accordingly, the two conditions directly guide the design of the proposed representations: condition (b) motivates the centering projection introduced in § 3.2.1, whereas condition (a) motivates the angular correction under class expansion described in § 3.3.

Hyperspherical Simplex Representations

We now formalize two hyperspherical simplex representations, which are derived directly from the softmax outputs and logits of a trained classification model.

Features Derived From Softmax Outputs

Given a model at update step $k$, the softmax output $\sigma(z^k)$, obtained by applying the softmax function $\sigma(\cdot)$ to the logits $z^k$, represents a probability distribution over the $C^k$ classes and lies within the probability simplex, which is defined as follows:

For a training sample of class $c$, the cross-entropy loss is minimized over the probability simplex $\Delta^{C^k-1}$ at the vertex $\mathbf{e}_c$, which corresponds to the one-hot encoded target label. Hence, the optimization of the cross-entropy loss drives the softmax outputs toward the vertices $\mathbf{e}_c$ of $\Delta^{C^k-1}$ during training (see Figure 1). A straightforward way to obtain hyperspherical representations from softmax outputs is to apply $\ell_2$-normalization directly. However, this approach does not satisfy condition (b) of § 3.1.

Since the probability simplex $\Delta^{C^k-1}$ lies entirely in the non-negative orthant, with its center $\mathbf{o}^k$ offset from the origin, any two softmax outputs have a non-negative inner product: the vertices $\mathbf{e}_c$ toward which the softmax outputs are driven are mutually orthogonal, and the $\ell_2$-normalized softmax outputs of different classes are separated by angles of at most $\pi/2$.

To overcome this limitation and obtain angles greater than $\pi/2$ between distinct classes, we apply a projection matrix $\mathbf{P}_{k,k}$ before the $\ell_2$-normalization of the softmax outputs. This projection centers the probability simplex $\Delta^{C^k-1}$ at the origin, mapping each softmax output to the corresponding point of a $(C^k-1)$-regular simplex⁴, while preserving pairwise distances, as shown in Figure 1.

To formulate the projection matrix, we define the class prototypes $\mathbf{v}_c^k$ as the vectors from the center $\mathbf{o}^k$ to the vertices $\mathbf{e}_c$ toward which the softmax outputs align:

These prototypes span the $(C^k-1)$-dimensional hyperplane of the regular simplex and therefore define the projection matrix. Accordingly, the deterministic projection $\mathbf{P}_{k,k}$ is defined as the following $C^k\times C^k$ matrix:

The hyperspherical simplex representation $h^k$ is then obtained as the $\ell_2$-normalized projection of the softmax output$^3$:

The class prototypes of the resulting representation $h^k$ satisfy condition (b) by construction, as they form a regular simplex centered at the origin with pairwise angles greater than $\pi/2$ [73]. Moreover, since the class prototypes $v_c^k$ are defined with respect to the canonical basis vectors $e_c$ associated with the one-hot encoding of class $c$, their directions are fixed for a given set of classes, so that condition (a) holds across model updates when the class set is unchanged; the case in which new classes are introduced during an update is addressed in § 3.3. On the hypersphere, we model each class distribution by a vMF distribution, the hyperspherical analogue of an isotropic Gaussian, whose mean direction is aligned with the class prototype (see Figure 1).

Features Derived From Logits

The simplex geometry induced by softmax cross-entropy training is reflected not only in the softmax outputs but also in the underlying logits. Motivated by the UFM analyses of Mixon et al. [64] and Fang et al. [31], we analyze this relationship directly at the level of the logits. Specifically, rather than modeling the shared backbone and classifier, we treat the logit vector of each training sample as an independent optimization variable, initialized at the origin (i.e., at the equiprobable softmax distribution), and study its evolution under gradient descent on the softmax cross-entropy loss; we refer to this abstraction as the logit-UFM. This formulation isolates the geometry induced directly by the loss from the additional interactions introduced by the network architecture and shared model parameters, leading to the following proposition:

This behavior also emerges in standard networks, as illustrated in Figure 2 for a representative three-class case from the CIFAR-10 dataset. At the end of training, the softmax outputs concentrate near the vertices of the probability simplex (colored area), while the logits exhibit a regular-simplex geometry (gray area) aligned with the same class prototype directions $v_c^k$. Moreover, the logits are expected to be approximately centered in the terminal phase of training, consistent with theoretical analyses of Neural Collapse (NC) [70] that yield centered prediction vectors at global minimizers (Hong & Ling, 2024) and with the observation that normalization mechanisms commonly used in neural networks tend to center latent representations [65]. We therefore directly $\ell_2$-normalize the logits, without the centering projection required for the softmax outputs, yielding a second hyperspherical simplex representation:

Analogously to the representation in Eq. (5), we model each class distribution of the normalized logit representation with a vMF distribution [5] whose mean direction is aligned with the class prototype $\mathbf{v}_c^k$. Since both representations share the same prototype directions, the representation in Eq. (6) inherits the geometric properties discussed in § 3.2.1 and can likewise be analyzed within the theoretical framework of Theorem 3.6. Compared with the representation derived by the softmax outputs as in Eq. (5), the normalized logit representation exhibits a larger angular spread (see Figure 2), preserving greater within-class variability, a property that has been associated with improved transferability and robustness of learned representations [22] (Wang & Isola 2020).

To assess whether the behavior predicted by Prop. 3.3 emerges in standard neural networks, and to quantify the spread of the two representations, we leverage the NC hypotheses introduced by Papyan et al. [70]. These hypotheses provide a methodology for examining how the representations derived from softmax outputs and logits collapse toward the class prototype directions defined in Eq. (3). The NC hypotheses evaluate whether the within-class covariance tends to 0 (NC1); whether class means form a simplex whose vertices have the same norm (NC2 equinorm) and are maximally and equally distant from each other (NC2 equidistance); whether class means converge to class prototypes (NC3); and whether the nearest class-mean rule for classifying features holds (NC4).

Figure 3 reports the corresponding NC measures for softmax outputs (blue curves) and logits (orange curves) during the terminal phase of training of a ResNet-18 on CIFAR-100. The higher NC1 values observed for logits indicate greater residual within-class variability than for softmax outputs. Similarly, the higher NC2-equidistance and NC3 values indicate a lower degree of convergence of the logit representations toward the ideal simplex geometry and the corresponding class-prototype directions. These results show that, while both representations exhibit the same underlying simplex structure, logits retain a larger within-class spread and a lower degree of prototype alignment than softmax outputs. This distinct trade-off between alignment and spread is reflected in the retrieval performance reported in § 4. Similar trends across different model architectures and datasets are reported in Appendix § 8.

Deterministic Angular Correction Under Class Expansion

In this section, we address the misalignment that arises in the proposed hyperspherical simplex representations when new classes are introduced during model updates, analyzing the behavior of the features through their class prototypes [70]. We formally characterize this misalignment and introduce a deterministic linear transformation that restores prototype alignment across updates.

As follows from Def. 3.2, the geometry of the probability simplex depends explicitly on the number of classes. When the model is updated from step $k$ to step $t$ by introducing additional classes, the classifier expands from $C^k$ to $C^t$ outputs, changing the simplex center and, consequently, the directions of the prototypes associated with the previously existing classes according to Eq. (3). Since the representations at the two steps have different dimensions, throughout this section vectors at step $k$ are embedded into $\mathbb{R}^{C^t}$ by zero-padding whenever they are compared with vectors at step $t$. As illustrated by the dashed arrows in Figure 4, the corresponding prototypes at step $t$ belong to a higher-dimensional simplex and are misaligned with their counterparts at step $k$. The resulting angular misalignment (derived in Appendix § 9) is the same for all previously existing classes and is given by:

where $C^{k}$ and $C^{t}$ represent the number of classes at steps $k$ and $t$, respectively, with $C^{k} \le C^{t}$. Eq. (7) shows that $\theta_{c}^{k\to t}=0$ when $C^{t}=C^{k}$, so that the corresponding class prototypes are aligned. In contrast, $\theta_{c}^{k\to t}>0$ when $C^{t}>C^{k}$, so that adding new classes always introduces a misalignment.

To align the prototypes when new classes are added, we compensate for the angular misalignment in Eq. (7) using a deterministic linear transformation, defined as the matrix $\mathbf{P}_{t,k}=[\mathbf{P}_{k,k}\mid\mathbf{0}]$, where $\mathbf{0}$ is the $C^{k}\times(C^{t}-C^{k})$ zero matrix.

The block $\mathbf{P}_{k,k}$, i.e., the centering projection in Eq. (4), acts on the first $C^{k}$ coordinates and maps the prototypes at step $t$ of the classes already present at step $k$, $\{\mathbf{v}_{c}^{t}\}_{c=1}^{C^{k}}$, onto the subspace spanned by their counterparts $\{\mathbf{v}_{c}^{k}\}_{c=1}^{C^{k}}$ at step $k$, thereby removing the angular misalignment introduced by the new classes. The zero block $\mathbf{0}$ discards the components along the $C^{t}-C^{k}$ coordinates introduced at step $t$, which have no counterpart in the step-$k$ representation. The following two propositions formalize this behavior.

Together, Prop. 3.4 and Prop. 3.5 explain the origin of the misalignment and how $\mathbf{P}_{t,k}$ removes it. Prop. 3.4 shows that the prototypes associated with newly introduced classes at step $t$ lie in directions orthogonal to the subspace spanned by the prototypes at step $k$. Since the step-$t$ prototypes remain centered at the origin, the updated prototypes of the existing classes acquire components along these orthogonal directions, resulting in the angular misalignment $\theta_{c}^{k\to t}$ defined in Eq. (7). Prop. 3.5 then shows that $\mathbf{P}_{t,k}$ removes these components and maps each updated old-class prototype onto the direction of its corresponding prototype at step $k$. Figure 4 illustrates this process: dashed arrows indicate the prototype change induced by class expansion, whereas solid arrows show how $\mathbf{P}_{t,k}$ restores the alignment.

Since $\mathbf{P}_{t,k}$ is determined entirely by the simplex geometry and the number of classes at the two update steps, it can be computed without learning additional parameters or accessing any data. It therefore provides a deterministic mechanism for restoring prototype alignment of the hyperspherical simplex representations in Eq. (5) and Eq. (6) across model updates under class expansion.

We refer to the approach based on the hyperspherical simplex representation derived from softmax outputs in Eq. (5) as Probability Simplex Projection (PSP), and to the approach based on the logit representation in Eq. (6) as Logits Simplex Projection (LSP). When $C^{t} > C^{k}$, the transformation $\mathbf{P}_{t,k}$ is applied to the softmax outputs or logits of the updated model before $\ell_{2}$-normalization, so that the resulting features lie in the representation space of step $k$; when $C^{t} = C^{k}$, the representations in Eq. (5) and Eq. (6) are used directly. Appendix F provides pseudocode for gallery indexing with PSP or LSP and for comparing query and gallery representations across model updates.

Theoretical Analysis of the Backward-Compatibility Definition

In the following, we establish geometric conditions under which the backward-compatibility inequalities hold in expectation, and then analyze whether these conditions are satisfied by the proposed hyperspherical simplex representations.

The vMF distribution in assumption (i) is employed as a tractable modeling assumption rather than as an exact characterization of the learned distributions: it yields closed-form expressions for the expected cosine distances in Eq. 1a and Eq. 1b in terms of the prototype directions and concentration parameters (Lemma G.1 in Appendix § 13), which enables the analytical characterization in Theorem 3.6. It provides a standard probabilistic model for directional data on the unit hypersphere (Banerjee et al., [5]), and hyperspherical and angular representations have been extensively investigated in deep classification and face recognition (Liu et al., [54]; Deng et al., 2017; Deng et al., [24]). More broadly, vMF-based formulations have been employed to define hyperspherical latent-variable models (Davidson et al., [23]), learn discriminative hyperspherical representations (Hasnat et al., [38]; Scott et al., [87]), model normalized contrastive and hyperspherical features (Du et al., [26]; Bai et al., [4]), and characterize class-conditional latent structure in theoretical analyses of cross-entropy learning (Reizinger et al., [80]). Remark 2 further clarifies the specific role of the vMF assumption and its first-moment characterization in the proof of Theorem 3.6.

Assumption (ii), that class representations become increasingly concentrated around their prototypes across model updates, is supported by several complementary observations. Scaling-law studies show that increasing model capacity or training data generally improves predictive, transfer, and retrieval performance (Fang et al., [32]; Prato et al., [75]; Zhai et al., [113]), motivating the model-update scenarios considered here, in which updated models are more capable or trained on larger datasets. Neural Collapse analyses provide a more direct geometric connection: within-class variability progressively decreases during training as features concentrate around their class means (Papyan et al., [70]; Yaras et al., [112]; Jiang et al., [48]), and stronger collapse with reduced relative within-class variability is observed as network width and depth increase (Yaras et al., [112]; Sükeník et al., [93]). Under the vMF modeling assumption, such a reduction in angular dispersion around the class prototype corresponds to an increase in the concentration parameter $\kappa$.

These observations primarily concern penultimate-layer features, but they can be related to the output representations used by PSP and LSP. For a linear classifier, variations in the backbone representations induce bounded variations in the corresponding logits (Gouk et al., [36]), and the progressive alignment between class means and classifier weights observed during training (Papyan et al., [70]) relates the geometry of the penultimate-layer representations to that of the logits. In Appendix H, we provide empirical evidence on CIFAR-10 that both logits and softmax outputs become increasingly concentrated around their class prototypes across model updates.

The aligned regular-simplex geometry centered at the origin, on which the proposed representations are built, satisfies the geometric conditions of Theorem 3.6. The following corollary formalizes this result for any hyperspherical representation exhibiting this geometry.

This follows from the geometry of a regular simplex centered at the origin, where the inner product between any two distinct normalized class prototypes is negative, implying a pairwise angle greater than $\pi/2$ [73]. Hence, once corresponding class prototypes are aligned across updates, both geometric conditions required by Theorem 3.6 hold. Remark 3 in Appendix G further clarifies the relation between this geometry and the Simplex ETF configuration arising in Neural Collapse.

The proposed PSP and LSP representations satisfy the geometric hypotheses of Corollary 3.7. Their class prototypes form a regular simplex centered at the origin: for PSP, by construction (Sec. 3.2.1); for LSP, exactly under the logit-UFM assumptions of Prop. 3.3, with the emergence of this geometry in neural networks assessed empirically through the Neural Collapse analysis of § 3.2.2. Corresponding prototypes remain aligned across updates when the class set is unchanged, while under class expansion the transformation $\mathbf{P}_{t,k}$ restores their alignment (Prop. 3.5). Therefore, under the assumptions of Theorem 3.6, PSP and LSP satisfy the compatibility inequalities of Eq. 1a and Eq. 1b in expectation. In Appendix I, we empirically isolate the two operations underlying PSP in a class-expansion update, showing that positive average margins for both compatibility inequalities are obtained only when the centering projection and the angular correction are combined.

The theoretical result is not specific to PSP and LSP: Corollary 3.7 also applies, for instance, to representations learned with pre-allocated $d$-simplex fixed classifiers [9, 11, 12], whose class prototypes are fixed a priori and thus remain aligned across updates. From a practical perspective, however, PSP and LSP do not require a shared pre-allocated fixed classifier across model updates.

Scalability via Top-$k$ Sparsification

Hyperspherical simplex representations derived from softmax outputs (Eq. (5) for PSP) or logits (Eq. (6) for LSP) have dimensionality equal to the number of classes. Although training on a larger number of classes can improve the quality and transferability of learned representations [120, 41], this class-dependent dimensionality poses a limitation in large-scale retrieval scenarios: the memory required to store the gallery scales linearly with both the number of gallery samples and the number of classes, potentially becoming prohibitive at scale. To address this, we construct sparse representations by retaining only the highest-ranked components of each feature vector, a choice motivated by the top-$k$ calibration of the softmax cross-entropy loss and the rank-preserving property of the softmax function [52, 111].

Following the theoretical analysis of Lapin et al. [52], the Bayes-optimal classifier under the softmax cross-entropy loss satisfies:

where $\alpha> 0$ is a class-independent constant and $p_y(x) = \mathbb{P}(Y = y \mid X = x)$ denotes the posterior probability of label $y$ given input $x$. Since the logarithm is strictly increasing, the optimal scores are a monotonic reparameterization of the class posterior probabilities and preserve their ordering, which underlies the top-$k$ calibration of the softmax cross-entropy loss [52]. Moreover, the softmax function is rank-preserving [111], i.e., $z_i > z_j \Longleftrightarrow\sigma(z_i) > \sigma(z_j)$, so that logits and softmax outputs induce the same class ranking. This ranking is also preserved by the operations defining PSP and LSP: the centering projection $\mathbf{P}_{k,k}$ subtracts the same constant from all entries of a softmax output, and $\ell_2$normalization rescales all entries by a positive factor. Consequently, the largest entries of PSP and LSP correspond to the highest-ranked classes.

Accordingly, we adopt a top-$k$ sparsification strategy for both PSP and LSP, applied after $\ell_2$-normalization: we retain the $k$ largest entries of the normalized feature vector $\mathbf{h}^k$ and set the remaining entries to zero. The resulting sparse representations can be directly processed by approximate nearest-neighbor search libraries designed for sparse high-dimensional data (e.g., PySparNN (Facebook Research, [30]) and Seismic (Bruch et al., [14])), supporting efficient search over large galleries (simhadri et al., 2025). In § 4, we empirically evaluate top-$k$ sparsification for both PSP and LSP, showing that high levels of sparsity can be achieved with negligible degradation in retrieval performance while preserving backward-compatibility.

Experimental Results

Building on the theoretical analysis, in this section we examine whether the backward-compatibility properties of PSP and LSP are reflected empirically across various model-update scenarios, under established training protocols and retrieval-based evaluation procedures.

Evaluation Setting

Following prior work on model compatibility (Shen et al., [88]; Zhang et al., [115]; Meng et al., [62]; Jaeckle et al., [46]), we adopt image retrieval as the evaluation task, where the goal is to retrieve images from a gallery $\mathcal{G} = \{(x_i,y_i)\}_{i=1}^{N_g}$ using queries from a query set $\mathcal{Q} = \{(x_i,y_i)\}_{i=1}^{N_q}$. Since verifying the inequalities of Def. 3.1 is impractical at scale (§ 3.1), we follow standard practice and adopt the Empirical Compatibility Criterion introduced by Shen et al. [88]:

where $\phi^k$ denotes the model at step $k$ and $M$ denotes a retrieval performance metric. The term $M(\phi^t,\phi^k;\mathcal{Q},\mathcal{G})$ denotes the cross-model accuracy, obtained when query representations are extracted using the updated model $\phi^t$ and gallery representations using the previous model $\phi^k$. Conversely, $M(\phi^k,\phi^k;\mathcal{Q},\mathcal{G})$ denotes the same-model accuracy, obtained when both query and gallery representations are extracted using the same model $\phi^k$.

Following prior work (Ramanujan et al., [79]; Zhou et al., [119]; Ricci et al., [81]; Jaeckle et al., [46]; Ricci et al., [82]), we use Cumulative Matching Characteristic (CMC@1) as the retrieval performance metric $M$, and the full test set of each dataset is used as both the query set and gallery set. Retrieval is performed using cosine distance between query and gallery representations, excluding the query sample itself from the gallery to avoid a trivial match. CMC@1 is computed as the fraction of queries for which the gallery sample ranked first has the same class label as the query.

To evaluate compatibility across multiple model updates, we construct a compatibility matrix following Biondi et al. [9]; Bui et al. [17]:

This compatibility matrix summarizes retrieval performance across all valid pairs of model updates, extending the evaluation beyond the two or three update steps commonly considered in previous studies (Shen et al., [88]; Zhou et al., [119]; Zhang et al., [114]). Compatibility is assessed by comparing each cross-model entry $\mathbf{C}_{t,k}$, with $t > k$, against the corresponding same-model entry $\mathbf{C}_{k,k}$ according to the Empirical Compatibility Criterion in Eq. (9).

To summarize the overall behavior across multiple updates, we adopt the metrics defined by Biondi et al. [9] and Biondi et al. [12], which aggregate the entries of the compatibility matrix into distinct measures, conceptually analogous to metrics used in continual learning (Lopez-Paz & Ranzato, [56]):

• Average Compatibility (AC), which measures the fraction of compatible cross-model comparisons over $T$ update steps: $AC = \frac{2}{T(T-1)} \sum_{t=2}^{T} \sum_{k=1}^{t-1} \mathbb{1}(C_{t,k} > C_{k,k})$, where $\mathbb{1}$ denotes the indicator function.

• Average Accuracy (AA), which measures the average retrieval performance across all same-model and cross-model comparisons over $T$ update steps, thus capturing the overall retrieval accuracy regardless of compatibility: $AA = \frac{2}{T(T+1)} \sum_{t=1}^{T} \sum_{k=1}^{t} C_{t,k}$.

• Average Compatibility Accuracy (ACA), which jointly captures retrieval performance and compatibility by averaging the cross-model accuracy over all update pairs, while assigning zero contribution to incompatible comparisons: $ACA = \frac{2}{T(T-1)} \sum_{t=2}^{T} \sum_{k=1}^{t-1} C_{t,k}\mathbb{1}(C_{t,k} > C_{k,k})$.

Standard Compatibility Benchmarks

In this section, we compare PSP and LSP with established backward-compatibility methods on standard compatibility benchmarks. Following prior work [88, 119, 62, 9, 69], we evaluate three update scenarios: extended classes, where new classes are introduced across model updates; extended backbone, where the model architecture is upgraded; and extended classes and backbone, where both changes occur simultaneously.

Following [88] and subsequent literature, unless otherwise specified, each updated model is randomly initialized and trained from scratch rather than initialized from the previous model. This protocol removes parameter inheritance across model updates, ensuring that observed compatibility cannot be attributed to warm-starting from a previously learned representation. Models are trained using classification as the surrogate task for representation learning. Implementation details for the standard compatibility benchmarks are provided in Appendix § 16.

We compare against the following backward-compatibility methods: BCT [88], RACT [114], AdvBCT [69], LCE [62], and CoReS [9]. We also include a Baseline based on the $\ell_2$-normalized encoder features without any compatible method applied. When the encoder feature dimensionality differs across model updates, the lower-dimensional representation is zero-padded to match the dimensionality of the higher-dimensional one before computing cross-model similarities. For reproducibility, we use the official publicly available implementations of all compared methods. All experiments are conducted on two NVIDIA A100 40 GB GPUs.

Extended Classes Benchmark

In the extended-classes benchmark, the training set of each dataset is partitioned into $T$ sequential tasks, each introducing a disjoint and equal number of new classes. At each update step $t$, the model is trained from scratch using the cumulative training data from tasks $1$ through $t$. Following [119];

Ricci et al. [81]; Jaeckle et al. [46]; Zhou et al. [118], retrieval is evaluated at each update step over the entire test set, including classes that have not yet been observed during training. This results in a semi-open-set evaluation scenario, in which the training class set expands across updates while retrieval is performed over the complete set of classes. Table 1 reports $AC$, $AA$, and $ACA$ on CIFAR-100 (Krizhevsky, [50]) using a ResNet-18 backbone. Table 2 reports the same metrics on larger-scale datasets: ImageNet-1K (Russakovsky et al., [85]) using a ResNet-50 backbone and Google Landmarks v2 (Weyand et al., [102]) using a ResNet-18 backbone initialized from ImageNet-1K pretrained weights at every update step. Overall, Table 1 and Table 2 show that PSP and LSP achieve competitive retrieval performance while consistently providing strong compatibility across model updates. The advantage of our proposed representations over the compared approaches becomes increasingly pronounced as the number of update steps increases, as reflected by the compatibility metrics in the 50-step setting on CIFAR-100 and the 5-step setting on ImageNet-1K. The compatibility matrices for the 5-step CIFAR-100 experiment are reported in Figure 9 in Appendix § 17, providing a visual summary of retrieval performance across model updates.

Method	2 steps			5 steps			20 steps			50 steps
	AC	AA	ACA	AC	AA	ACA	AC	AA	ACA	AC	AA	ACA
Baseline	0	29.63	0	0	13.60	0	0	04.22	0	0	02.29	0
BCT	1	40.64	35.51	0	31.03	0	0	22.66	0	0.01	15.78	0.84
RACT	1	44.19	45.09	0.60	30.43	16.53	0.05	21.61	2.42	0	16.71	0
AdvBCT	0	35.32	0	0.40	26.13	11.67	0.02	19.79	0.19	0	14.70	0.03
LCE	1	43.48	40.71	0.10	32.63	04.63	0	20.95	0.25	0	13.81	0.03
CoReS	1	40.94	35.21	0	29.63	0	0.12	24.88	4.61	0.11	23.50	3.91
PSP	1	36.31	29.05	0.90	26.04	21.76	0.52	20.56	12.99	0.39	19.42	10.76
LSP	1	41.14	36.38	0.70	30.36	21.91	0.44	24.11	13.26	0.36	22.68	11.25

Method	ImageNet-1K, 2 steps, $AC$	ImageNet-1K, 2 steps, $AA$	ImageNet-1K, 2 steps, $ACA$	ImageNet-1K, 5 steps, $AC$	ImageNet-1K, 5 steps, $AA$	ImageNet-1K, 5 steps, $ACA$	Google Landmark v2, 2 steps, $AC$	Google Landmark v2, 2 steps, $AA$	Google Landmark v2, 2 steps, $ACA$	Google Landmark v2, 5 steps, $AC$	Google Landmark v2, 5 steps, $AA$	Google Landmark v2, 5 steps, $ACA$
Baseline	0	37.52	0	0	16.18	0	0	07.66	0	0	07.87	0
BCT	1	48.81	45.86	0.20	34.96	5.28	1	09.37	08.22	0	08.46	0
RACT	1	55.17	51.16	0.10	40.94	6.20	0	11.01	0	0	09.99	0
AdvBCT	0	53.07	0	0	43.91	0	1	11.12	09.67	0	10.73	0
LCE	0	47.22	0	0	32.67	0	1	09.41	08.03	0.40	10.31	3.80
CoReS	0	46.11	0	0	37.67	0	-	-	-	-	-	-
PSP	1	49.10	39.80	1	35.20	31.09	1	08.74	08.24	0.90	08.53	7.21
↪ w top-256	1	49.11	39.81	1	35.20	31.09	1	08.74	08.24	0.90	08.53	7.21
LSP	1	50.72	45.62	0.50	38.03	14.28	1	09.31	08.36	0.80	09.36	7.34
↪ w top-256	1	50.34	45.84	0.70	37.96	24.85	1	09.30	08.35	0.80	09.34	7.31

In the 2-step setting, where compatibility is evaluated over a single model-update pair, several of the compared approaches achieve strong performance. On CIFAR-100, BCT, RACT, LCE, CoReS, PSP, and LSP all satisfy the empirical compatibility criterion ($AC = 1$), with RACT achieving the highest retrieval performance among the compared methods. A similar behavior is observed on ImageNet-1K, where BCT, RACT, PSP, and LSP satisfy the compatibility criterion, while RACT obtains the highest $AA$ and $ACA$. On Google Landmarks v2, BCT, AdvBCT, LCE, PSP, and LSP satisfy the criterion, whereas RACT does not. In contrast, the Baseline based directly on the encoder features does not achieve compatibility in any evaluated setting, showing that independently trained encoder representations are not naturally compatible across model updates.

As the number of updates increases, the differences among the methods become more pronounced. On CIFAR-100, the compatibility of BCT, RACT, AdvBCT, and LCE drops to nearly zero in the 20- and 50-step settings. CoReS is more robust than these pairwise compatibility approaches over longer update sequences and retains comparatively high $AA$; however, its $AC$ and $ACA$ remain substantially below those of PSP and LSP. For example, in the 50-step setting, CoReS achieves $AC = 0.11$ and $ACA = 3.91$, compared with $AC = 0.39$ and $ACA = 10.76$ for PSP and $AC = 0.36$ and $ACA = 11.25$ for LSP. A similar trend is observed in the 5-step ImageNet-1K benchmark: although several competing methods retain high $AA$, their compatibility frequency is low or zero, whereas PSP achieves $AC = 1$ and $ACA = 31.09$, and LSP achieves $AC = 0.50$ and $ACA = 14.28$. On Google Landmarks v2, PSP and LSP similarly attain the highest compatibility values in the 5-step setting, with $AC = 0.90$ and $AC = 0.80$, respectively, compared with $AC = 0.40$ for LCE and $AC = 0$ for the remaining evaluated baselines.

One possible explanation for the degradation observed for several of the compared methods as the number of updates increases is that regularization- and mapping-based approaches typically establish compatibility only with respect to the previous model. When such compatibility is enforced sequentially across multiple updates, deviations and misalignment with respect to earlier model versions may accumulate because compatibility with all previous representations is not directly enforced. Rather than sequentially enforcing compatibility with their predecessors, PSP and LSP exploit a regular-simplex geometry that provides a common reference across all model updates, consistently with Corollary 3.7.

Among the compared approaches, CoReS [9] is conceptually closer to PSP and LSP, as it also exploits a regular-simplex geometry as reference across model updates. The main practical distinction is that CoReS requires the classifier geometry and the number of prototypes to be pre-allocated before training, including prototypes reserved for classes that may be introduced in future updates [74, 9]. The outputs reserved for future classes participate in the cross-entropy objective from the beginning of training and act as negative classes for current samples [74, 9], thereby constraining representation learning with respect to a prototype geometry and class capacity specified in advance. This fixed pre-allocated prototype assignment may limit adaptation to the observed class structure (Saadabadi et al., 2024). In contrast, PSP and LSP derive their reference geometry directly from the output space of each model. Furthermore, CoReS scales poorly with the number of classes, as its memory requirements, driven by both the pre-allocated fixed classifier and the additional linear layer mapping the backbone features to the simplex representation space, grow substantially with the class count. For Google Landmarks v2, these requirements made training computationally infeasible under our experimental configuration, and CoReS results are therefore not reported in Table 2.

Notably, across the different numbers of update steps reported in Table 1 and Table 2, LSP consistently achieves higher $AA$ than PSP, while its $AC$ is generally equal to or lower than that of PSP. This behavior is consistent with the larger within-class spread and lower prototype alignment of LSP observed in the NC analysis of § 3.2.2. As a result of this trade-off, LSP often achieves comparable or higher $ACA$. However, since incompatible comparisons contribute zero to $ACA$, PSP achieves a higher $ACA$ when it is compatible in a substantially larger fraction of update pairs, as in the 5-step ImageNet-1K setting ($AC = 1$ for PSP vs. $AC = 0.50$ for LSP).

The rows labeled “$\rightarrow$ w top-256” in Table 2 report the performance of PSP and LSP after applying ent top-$k$ values in terms of $AC$, $AA$, and $ACA$, in the top-$k$ sparsification described in § 3.5. Top-256 sparsification leaves the PSP results essentially unchanged across the evaluated benchmarks, while also preserving the retrieval accuracy of LSP. In the 5-step ImageNet-1K setting, sparsification increases the $AC$ of LSP from 0.50 to 0.70 and its $ACA$ from 14.28 to 24.85, with only a negligible change in $AA$ (from 38.03 to 37.96). Consistently, Table 3 shows that retrieval accuracy remains essentially stable over the evaluated range of $k$, while the observed compatibility behavior is preserved.

$Top$-$k$ with $k$	PSP $AC$	PSP $AA$	PSP $ACA$	LSP $AC$	LSP $AA$	LSP $ACA$
256	1	35.20	31.09	0.7	37.96	24.85
384	1	35.20	31.09	0.7	38.00	24.87
512	1	35.20	31.09	0.5	38.03	14.24
768	1	35.20	31.09	0.5	38.01	14.28
896	1	35.20	31.09	0.5	38.03	14.28
1000	1	35.20	31.09	0.5	38.03	14.28

The memory benefit of sparsification becomes particularly relevant when the number of classes is large. For a gallery of $N_g$ representations of dimension $C$, dense FP32 storage requires approximately $4N_gC$ bytes, whereas storing the top-$k$ representation using one 32-bit value and one 32-bit index per retained coordinate requires approximately $8N_gk$ bytes, excluding minor sparse-format metadata overhead. For Google Landmarks v2, where the representation dimensionality exceeds 81k at the final update, this corresponds to approximately 318 KiB per dense representation and only 2 KiB with $k = 256$, yielding approximately a 159× reduction in gallery storage. Sparse representations also avoid dense $C$-dimensional similarity computation; the resulting search-time benefit depends on the adopted indexing and search backend (simhadri et al., 2025).

Extended Backbone Benchmark

In this experimental setting, we maintain the same dataset across all update steps (with no introduction of additional classes during model updates) while increasing model architectural complexity across two steps (i.e., ResNet-18 $\to$ ResNet-50). The goal is to evaluate compatibility when model updates involve a change in model architecture rather than the introduction of new training classes. Results for CIFAR-100 and ImageNet-1K are summarized in Table 4. Most of the evaluated approaches satisfy the compatibility criterion ($AC = 1$), suggesting that compatibility is generally easier to maintain in this setting than in the extended-classes benchmark. Since this benchmark consists of two update steps, $AC$ is determined by a single cross-model comparison, and performance differences therefore emerge primarily in terms of $AA$ and $ACA$. LSP achieves the highest $AA$ and $ACA$ on CIFAR-100 and remains within 0.1–0.2 percentage points of the best-performing methods, CoReS and RACT, on ImageNet-1K. These results indicate that LSP maintains strong compatibility and retrieval performance even when the model architecture changes substantially. Notably, PSP and LSP achieve competitive performance without requiring compatibility-specific regularization, learnable mappings, or modifications to the model architecture.

Method	Ext. Backbone						Ext. Classes & Backbone
	CIFAR-100			ImageNet-1K			CIFAR-100			ImageNet-1K
	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$	$AC$	$AA$	$ACA$
Baseline	0	35.13	0	0	40.58	0	0	29.88	0	0	35.88	0
BCT	1	53.57	54.06	1	61.80	61.43	1	43.61	40.79	1	50.64	42.98
RACT	1	53.80	54.62	1	62.15	62.07	1	43.66	41.01	1	52.74	48.92
AdvBCT	0	49.56	0	1	59.22	57.73	1	39.06	35.05	1	50.37	45.42
LCE	1	52.14	52.26	1	61.22	61.40	1	42.42	39.72	1	52.28	48.06
CoReS	1	56.67	57.43	1	62.03	63.19	1	42.61	37.20	1	49.10	39.90
PSP	1	54.57	56.01	1	68.35	65.28	1	38.65	32.84	1	48.42	41.24
LSP	1	56.76	57.96	1	62.00	63.16	1	43.08	37.89	1	52.13	48.26

Extended Classes and Backbone Benchmark

This experimental configuration, consisting of two update steps, combines the two previously examined update mechanisms: at the second step, the model is retrained from scratch on an expanded training set using a more expressive backbone. Results for this combined setting are reported in Table 4 for the CIFAR-100 and ImageNet-1K datasets. The Baseline, which does not employ any compatibility-preservation mechanism, fails to satisfy the compatibility criterion ($AC = 0$) on both datasets, consistent with the behavior observed in the previous experimental settings. In contrast, all evaluated backward-compatibility approaches, including PSP and LSP, satisfy the criterion ($AC = 1$), showing that compatibility can be maintained even when the model update simultaneously introduces new classes and changes the backbone architecture. Across both CIFAR-100 and ImageNet-1K, LSP shows robust performance under this combined update setting, achieving $AA$ and $ACA$ values that match or closely approach those of the strongest compared methods. PSP also satisfies the compatibility criterion on both datasets, but attains lower $AA$ and $ACA$ than LSP. This behavior is consistent with the smaller within-class spread of the softmax output representation observed in the Neural Collapse analysis of § 3.2, which may reduce retrieval performance in the semi-open-set setting, where the test set also contains classes not yet observed during training.

Compatibility Across Independently Trained Models From Public Repositories

The standard compatibility benchmarks of § 4.2 consider models explicitly trained within a backward-compatibility protocol. Here, we evaluate a complementary and more practical setting in which compatibility is measured directly between independently trained models obtained from a public repository, without any additional compatibility-specific training or adaptation.

We consider two sequences of pretrained models from the publicly available torchvision repository ([59]): a convolutional and hybrid sequence comprising AlexNet ([51]), ResNet-50 ([39]), RegNetX_3.2GF ([77]), ResNet-152 ([39]), and MaxViT_T ([97]), and a transformer sequence comprising ViT-B/32, ViT-B/16, and ViT-L/16 ([25]). All models are pretrained on ImageNet-1K while differing substantially in architecture and representational capacity.

We evaluate retrieval on the ImageNet-1K test set, which follows the pretraining distribution (closed-set), and on Places365 ([117]), whose classes and data distribution differ from those used for pretraining (open-set). We compare PSP and LSP with the Baseline introduced in § 4.2, obtained by $\ell_2$-normalizing the encoder features. Existing backward-compatibility methods are not included because they require additional training, adaptation, or architectural modifications and therefore cannot be directly applied to these publicly available, independently trained models.

The results in Table 5 show a clear advantage of the proposed hyperspherical simplex representations over the Baseline. PSP provides highly consistent compatibility across the evaluated model sequences, attaining $AC=1$ in three of the four settings and $AC=0.90$ in the remaining open-set setting. LSP also substantially improves compatibility over the Baseline, although it satisfies the empirical compatibility criterion for a smaller fraction of model-update pairs. The compatibility matrices in Figure 5 illustrate these differences: LSP and the Baseline achieve comparable same-model retrieval performance along the main diagonal, but the Baseline fails to maintain this performance across model updates, resulting in $AC=0$. A different trade-off between PSP and LSP emerges in closed-set and open-set evaluation. On ImageNet-1K, corresponding to the same data distribution used to pretrain all models, PSP consistently achieves higher $AC$, $AA$, and $ACA$ than LSP. This behavior is consistent with the stronger alignment of PSP representations with the regular-simplex geometry and their lower within-class spread discussed in § 3.2.2. On Places365, whose data distribution differs from the pretraining one, LSP instead achieves higher $AA$ than PSP for both model sequences. This result is consistent with the broader angular spread of LSP representations, which may provide greater flexibility under domain shift. However, PSP retains substantially higher compatibility in this setting, achieving $AC = 0.90$ and $1.00$, compared with $0.20$ and $0.33$ for LSP, and consequently also obtains higher $ACA$.

Method	AlexNet $\to$ R-50 $\to$ RegNet $\to$ R-152 $\to$ MaxViT — ImageNet-1K $AC$	AlexNet $\to$ R-50 $\to$ RegNet $\to$ R-152 $\to$ MaxViT — ImageNet-1K $AA$	AlexNet $\to$ R-50 $\to$ RegNet $\to$ R-152 $\to$ MaxViT — ImageNet-1K $ACA$	AlexNet $\to$ R-50 $\to$ RegNet $\to$ R-152 $\to$ MaxViT — Places365 $AC$	AlexNet $\to$ R-50 $\to$ RegNet $\to$ R-152 $\to$ MaxViT — Places365 $AA$	AlexNet $\to$ R-50 $\to$ RegNet $\to$ R-152 $\to$ MaxViT — Places365 $ACA$	ViT-B-32 $\to$ ViT-B-16 $\to$ ViT-L-16 — ImageNet-1K $AC$	ViT-B-32 $\to$ ViT-B-16 $\to$ ViT-L-16 — ImageNet-1K $AA$	ViT-B-32 $\to$ ViT-B-16 $\to$ ViT-L-16 — ImageNet-1K $ACA$	ViT-B-32 $\to$ ViT-B-16 $\to$ ViT-L-16 — Places365 $AC$	ViT-B-32 $\to$ ViT-B-16 $\to$ ViT-L-16 — Places365 $AA$	ViT-B-32 $\to$ ViT-B-16 $\to$ ViT-L-16 — Places365 $ACA$
Baseline	0	21.63	0	0	8.38	0	0	38.63	0	0	15.10	0
PSP	1	74.34	76.12	0.9	15.92	14.08	1	78.94	80.53	1	18.31	17.60
LSP	0.7	60.68	37.61	0.2	21.36	4.06	0.67	71.82	45.91	0.33	25.71	7.99

Overall, these experiments show that PSP and LSP can substantially improve cross-model retrieval compatibility even between models that were not explicitly trained to be compatible. PSP provides the most consistent empirical compatibility, whereas LSP preserves strong retrieval accuracy and achieves higher open-set $AA$, at the cost of lower compatibility than PSP.

These results reflect a structural property of the proposed representations: their common reference is determined by the label space rather than by any interaction between models. Models trained on the same set of classes under the standard class-indexing convention of a dataset, as is the case for all torchvision models trained on ImageNet-1K, induce the same one-hot encoding and therefore share the same reference geometry, without requiring coordination during training or access to previous models. Complementary compatibility matrices for the open-set evaluation and the ViT model sequence are reported in Appendix § 18.

Conclusions

In this paper, we introduce hyperspherical simplex representations derived from softmax outputs (PSP) and logits (LSP) for backward-compatible retrieval. The proposed representations leverage the simplex structure induced in the model output space by softmax cross-entropy training with one-hot encoded class labels, which provides a common geometric reference that aligns corresponding class prototypes across model updates, without requiring compatibility-specific losses, learned mapping functions, a shared pre-allocated fixed classifier, or modifications to the model architecture. When new classes are introduced, the resulting change in simplex geometry is compensated by a deterministic linear transformation that restores the alignment of the prototypes of previously observed classes. We provide a theoretical analysis of the formal backward-compatibility inequalities, showing that, under a von Mises–Fisher model with class concentration increasing across updates, they are satisfied in expectation if corresponding class prototypes are aligned across updates and prototypes of different classes form angles greater than $\pi/2$. Aligned regular-simplex representations centered at the origin satisfy both conditions; since PSP instantiates this geometry by construction and LSP under the logit-UFM, both satisfy the compatibility inequalities in expectation. We show that PSP and LSP achieve competitive retrieval performance on standard backward-compatibility benchmarks, with particularly high compatibility in scenarios involving many model updates. Experiments further demonstrate that the proposed representations improve cross-model retrieval compatibility between independently trained models obtained from a public repository, without any additional adaptation. Finally, we introduce a top-$k$ sparsification strategy motivated by the rank-preserving property of the softmax to address the linear growth of the representation dimensionality with the number of classes. Across the evaluated settings, top-$k$ sparsification substantially reduces representation storage, with negligible changes in retrieval accuracy and without degrading compatibility, supporting the use of PSP and LSP in large-scale retrieval scenarios.

Limitations. PSP and LSP rely on the simplex structure induced in the model output space by softmax cross-entropy training with one-hot encoded class labels. This requirement limits the direct applicability of our approach to models trained with alternative objectives, such as self-supervised pretraining without explicit classification heads or regression-based formulations. However, this constraint can be mitigated by training a linear classifier on top of a frozen pretrained encoder using a modest amount of external labeled data (see Appendix § 18.§ 1), with limited computational overhead since only the classifier is optimized.

A second limitation concerns the consistency of the label space across model updates: our method assumes that the mapping between each class label and its output index remains unchanged. Changes in class ordering or label definitions would disrupt the geometric correspondence on which the backward-compatibility of the proposed representations relies. A reordering of the classes can be undone by a fixed permutation of the output coordinates, provided that the correspondence between old and new indices is known. Moreover, model updates are commonly driven by additional samples for existing classes or by the introduction of new classes, corresponding to the domain-incremental and class-incremental scenarios of incremental learning [98, 53] and to the update settings considered in backward-compatible representation learning [88]; in both cases, the existing class indices are preserved.

Finally, our theoretical analysis relies on modeling assumptions, namely von Mises–Fisher class distributions and increasing concentration across updates, which provide a tractable approximation of learned representations rather than an exact characterization, so that the resulting guarantees hold in expectation under these assumptions. Empirically, PSP and LSP nonetheless substantially improve compatibility across the evaluated settings, indicating that the identified geometric conditions are informative in practice.

Broader Impact Statement

Backward-compatible retrieval can reduce the computational, storage, and environmental costs associated with repeatedly re-indexing large galleries. However, retaining legacy representations may also preserve biases or limitations inherited from earlier models, training data, or gallery composition. Moreover, aggregate compatibility metrics do not guarantee uniform performance across subpopulations. Future work should therefore consider disaggregated compatibility measures and establish when gallery re-indexing is preferable to preserving compatibility, particularly after updates intended to correct bias, modify class semantics, remove previously learned information, or address substantial distribution shifts.

Proof of Proposition 3.3

Proof. Consider a sample belonging to class $c$ at model update step $k$, with $1 \le c \le C^{k}$. For notational simplicity, let $C = C^{k}$, and let $\mathbf{z}^{(r)} \in\mathbb{R}^{C}$ denote its logit vector at optimization step $r$. We denote by $\mathbf{e}_{c}$ the corresponding one-hot encoded label vector.

Motivated by the UFM analyses of Mixon et al. [64]; Fang et al. [31], we consider a logit-space adaptation in which the shared backbone and classifier are not modeled explicitly. Instead, the logit vector associated with each training sample is treated as an independent unconstrained optimization variable and is optimized directly by gradient descent on the softmax cross-entropy loss. This idealized abstraction (logit-UFM) allows us to isolate the geometry induced by the loss from the additional coupling introduced by the network architecture and its shared parameters. Within this abstraction, we consider an equiprobability initialization by placing each logit vector at the origin of the logit space. Consequently, the corresponding softmax outputs are the uniform probability distribution i.e., $\mathbf{o}^{k} = [1/C^{k}, \ldots, 1/C^{k}]$. The gradient of the softmax cross-entropy loss with respect to the $j$-th logit is

Consequently, gradient descent updates the logit vector according to $\mathbf{z}^{(r+1)} = \mathbf{z}^{(r)} + \eta_{r}(\mathbf{e}_{c} - \sigma(\mathbf{z}^{(r)}))$, where $\eta_{r} > 0$ denotes the learning rate.

The first gradient update is

and hence

We now show by induction that this alignment is preserved throughout optimization. Suppose that, at iteration $r$, $\mathbf{z}^{(r)} = a_{r}\mathbf{v}_{c}^{k}$ for some scalar $a_{r} \ge0$. Here, $a_{r}$ is the scale of the logit vector along the prototype direction $\mathbf{v}_{c}^{k}$: it determines its norm, $\lVert\mathbf{z}^{(r)}\rVert_{2} = a_{r}\lVert\mathbf{v}_{c}^{k}\rVert_{2}$, and equals the margin $z_{c}^{(r)} - z_{j}^{(r)}$ between the target logit and each non-target logit $j \ne c$. At initialization, $a_{0} = 0$, and after the first update, $a_{1} = \eta_{0}$.

Since

all non-target components of $\mathbf{z}^{(r)}$ are equal and therefore have identical softmax probabilities.

Let

denote the target-class probability. Because the softmax probabilities sum to one, for every $j \ne c$,

Consequently,

Substituting this expression into the gradient update yields

Hence, $\mathbf{z}^{(r+1)}$ is also a non-negative scalar multiple of $\mathbf{v}_c^k$. Thus, by induction, the logit vector remains a non-negative scalar multiple of $\mathbf{v}_c^k$ at every optimization step. Since

is continuous and strictly increasing in $a_r$ on $[0,\infty)$, with $p_r\to1$ if and only if $a_r\to\infty$, the hypothesis that the softmax output converges to the vertex $\mathbf{e}_c$ (i.e. $p_r\to1$) forces $a_r\to\infty$. Throughout this entire trajectory—including in this limit—$\mathbf{z}^{(r)}=a_r\mathbf{v}_c^k$, so the logit direction remains aligned with

Convergence of PSP and LSP to a Simplex Configuration

In the following, we present additional experiments beyond those discussed in Figure 3, illustrating the NC metrics for softmax outputs (blue curves) and logits (orange curves) during training for ResNet-50 and DenseNet on the CIFAR-100 and Tiny-ImageNet-200 datasets.

The results in Figure 6 present plots that are consistent with those shown in Figure 3 for ResNet-18 on CIFAR-100, confirming that logits exhibit larger spread of feature distributions and less alignment with the simplex geometry than softmax outputs.

Mathematical Derivation of Eq. 6

Let $\mathbf{v}_{c}^{k}=\mathbf{e}_{c}-\mathbf{o}^{k}\in\mathbb{R}^{C^{k}}$ and $\mathbf{v}_{c}^{t}=\mathbf{e}_{c}-\mathbf{o}^{t}\in\mathbb{R}^{C^{t}}$ with $C^{k}\leq C^{t}$ the prototypes of class $c$ at two different steps $t$ and $k$. To compare prototypes across different dimensions, we extend $\mathbf{v}_{c}^{k}$ to $\mathbb{R}^{C^{t}}$ by zero-padding (not affecting the norm of a vector): $\tilde{\mathbf{v}}_{c}^{k}=[\mathbf{v}_{c}^{k}\mid0_{C^{t}-C^{k}}]\in\mathbb{R}^{C^{t}}$.

We compute the dot product between the two prototypes:

Since the norms of the two prototypes are:

their cosine similarity is:

Therefore, the misalignment angle between $\tilde{\mathbf{v}}_{c}^{k}$ and $\mathbf{v}_{c}^{t}$ is provided by the following formula:

Proof of Proposition 3.4

Proof. We first demonstrate the proposition in the case where only one class is added between step $k$ and step $t$. For simplicity in notation, we denote $C^{k}=d$, resulting in $C^{t}=C^{k}+1=d+1$. We assume that existing classes retain their coordinate indices across the model update. If $d=1$, the old prototype is zero and the proposition holds trivially; hence, we assume $d\geq2$ in the following calculation.

From Eq. (3), a prototype vector of the $c$-th class at step $k$ is denoted as $\mathbf{v}_{c}^{k}=\mathbf{e}_{c}-\mathbf{o}^{k}$ in $\mathbb{R}^{d}$, where $1\leq c\leq d$. Each component $v_{c,j}^{k}$ is:

Adding a new class in the training set at step $t$ adds a new vertex $\mathbf{e}_{d+1}$ in the probability simplex, shifting its center to: $\mathbf{o}^{t}=\frac{1}{d+1}\sum_{c=1}^{d+1}\mathbf{e}_{c}$. The prototype vector of the new class $d+1$ is then defined as $\mathbf{v}_{d+1}^{t}=\mathbf{e}_{d+1}-\mathbf{o}^{t}$.

To measure the angle between the new prototype and the old prototypes, we compute the cosine similarity between $\mathbf{v}_{d+1}^{t}$ and the generic prototype $\mathbf{v}_{c}^{k}$ after zero-padding the latter. According to Eq. (11), the norms of $\mathbf{v}^{k}_{c}$ and $\mathbf{v}^{t}_{d+1}$ are respectively:

and

The components of the corresponding normalized vectors are respectively:

and

In order to compute the dot product, $\mathbf{v}^{k}_{c}$ is padded with a zero component (this operation does not change its norm). Retaining the same notation for the embedded vector, we write:

Then, the dot product can be expressed as follows, with the components in the $c$-th and $(d+1)$-th positions written separately:

Since the component $v^{k}_{c,d+1}=0$, the expression in Eq. (16) simplifies to:

Using Eq. (14) and Eq. (15), we obtain:

This demonstrates that a generic prototype vector $\mathbf{v}^{k}_{c}$ and the new prototype vector $\mathbf{v}^{t}_{d+1}$ are orthogonal.

More generally, if $C^{t}=d+m$ with $m\geq1$, every new-class prototype $\mathbf{v}^{t}_{a}$, $a=d+1,\ldots,d+m$, has its first $d$ components equal to $-1/(d+m)$. After padding each old prototype with $m$ zeros, we obtain:

Since this holds for every $c=1,\ldots,d$, each new-class prototype is orthogonal to the subspace spanned by the zero-padded class prototypes at step $k$.

Proof of Proposition 3.5

Proof. Without loss of generality, we demonstrate the proposition in the case where only one class is added between step $k$ and step $t$. For simplicity in notation, we denote $C^{k}=d$, resulting in $C^{t}=C^{k}+1=d+1$.

Let $\mathbf{v}^{k}_{c}=\mathbf{e}_{c}-\mathbf{o}^{k}$ and $\mathbf{v}^{t}_{c}=\mathbf{e}_{c}-\mathbf{o}^{t}$ the prototype vectors of the $c$-th class at step $k$ and $t$, respectively. Their components $v^{k}_{c,j}$ and $v^{t}_{c,j}$ are respectively defined as:

and

We consider the vector $\mathbf{u}^{t}_{c}$ in $\mathbb{R}^{d}$ obtained from $\mathbf{v}^{t}_{c}$ in $\mathbb{R}^{d+1}$ using the linear transformation matrix $\mathbf{P}_{t,k}$:

where $\mathbf{P}_{t,k}=[\mathbf{P}_{k,k}\mid\mathbf{0}]$ with $\mathbf{P}_{k,k}=[\mathbf{v}^{k}_{c}]_{c=1}^{d}$ a $d\times d$ matrix and $\mathbf{0}$ a zero matrix $d\times1$. The following expressions hold for $u^{t}_{c,c}$ and $u^{t}_{c,j}$, respectively:

where the $c$-th component of $\mathbf{u}^{t}_{c}$ has been put into evidence in the first expression, while the second expression considers a generic component $j\ne c$. Substituting the expressions in Eq. (17) and Eq. (18) into Eq. (20) and Eq. (21) respectively, we obtain the following equations:

and

i.e., the components of the vector $u^{t}_{c}$ are identical to those of $v^{k}_{c}$.

It results that the vector obtained from $v^{t}_{c}$ is aligned with the prototype vector $v^{k}_{c}$ through the matrix $\mathbf{P}_{t,k}$:

This proof can be immediately extended to model updates where multiple classes are introduced.

Pseudocode

This appendix describes how PSP and LSP are integrated into a retrieval system. Importantly, PSP and LSP are two alternative feature definitions, not post-hoc transformations to be applied to arbitrary embeddings after deployment. Whether to use features derived from softmax outputs (PSP, Eq. (5)) or logits (LSP, Eq. (6)) must therefore be chosen before the first deployment of the retrieval system and then used consistently to build the gallery index and process future queries. Accordingly, gallery items are indexed once using the chosen representation and stored in the retrieval database.

The deterministic linear transformation $\mathbf{P}_{t,k}$ is required only when the updated model has a larger output space than the gallery model, i.e., when $C^{t}>C^{k}$. In this case, it maps the query representation produced at step $t$ into the $C^{k}$-dimensional feature space of the gallery indexed at step $k$. When $C^{t}=C^{k}$, no cross-update transformation is required. PSP queries are computed using the standard within-step projection $\mathbf{P}_{k,k}$, which is part of the PSP feature definition, whereas LSP queries are obtained by directly normalizing the logits. In the backward-compatible setting, the gallery is not re-indexed. This is the inference-time protocol used in our compatibility experiments.

Alg. 1 describes the initial gallery indexing procedure with PSP or LSP. Alg. 2 describes the query-time compatibility correction used only when features extracted at step $t$ must be compared against a gallery indexed at step $k$.

Alg. 1 is used to define and store gallery features when the retrieval system is first deployed. If, after a model update, the gallery is fully re-indexed, the same algorithm is simply applied again to all gallery items using the updated model and the same chosen representation type. By contrast, Alg. 2 is used only in the backward-compatible retrieval setting, where the gallery remains fixed at step $k$ and only the incoming queries are processed by the updated model at step $t$. In this case, PSP/LSP are not re-applied to the stored gallery features; only the query-side representation is corrected through $\mathbf{P}_{t,k}$ to enable comparison with the gallery without re-indexing.

We emphasize that the deterministic transformation $\mathbf{P}_{t,k}$ is not part of the generic feature-definition step of the retrieval system. It is only required when features extracted at different update steps with an increased number of classes must be compared. During training, there is no dependence on previous models, and no additional losses, alignment modules, or architectural modifications are introduced.

Algorithm 1  Initial Gallery Indexing with PSP or LSP
Require: Model at gallery step $k$ with $\mathcal{C}^{k}$ classes; gallery item $\mathbf{x}$; representation type repr $\in \{\mathrm{PSP}, \mathrm{LSP}\}$
Ensure: Stored gallery feature $\mathbf{h}^{k}(\mathbf{x})$
1: Compute logits $\mathbf{z}^{k}(\mathbf{x})$
2: if repr = PSP then
3:     Compute softmax outputs $\mathbf{p}^{k}(\mathbf{x}) \leftarrow \sigma(\mathbf{z}^{k}(\mathbf{x}))$
4:     $\tilde{\mathbf{h}}^{k}(\mathbf{x}) \leftarrow \mathbf{P}_{k,k}\mathbf{p}^{k}(\mathbf{x})$
5:     if $\|\tilde{\mathbf{h}}^{k}(\mathbf{x})\|_{2} > 0$ then
6:         $\mathbf{h}^{k}(\mathbf{x}) \leftarrow \tilde{\mathbf{h}}^{k}(\mathbf{x})/\|\tilde{\mathbf{h}}^{k}(\mathbf{x})\|_{2}$
7:     else
8:         $\mathbf{h}^{k}(\mathbf{x}) \leftarrow \mathbf{0}$
9: else if repr = LSP then
10:    if $\|\mathbf{z}^{k}(\mathbf{x})\|_{2} > 0$ then
11:        $\mathbf{h}^{k}(\mathbf{x}) \leftarrow \mathbf{z}^{k}(\mathbf{x})/\|\mathbf{z}^{k}(\mathbf{x})\|_{2}$
12:    else
13:        $\mathbf{h}^{k}(\mathbf{x}) \leftarrow \mathbf{0}$
14: Store $\mathbf{h}^{k}(\mathbf{x})$ in the gallery index

Algorithm 2  Query-Side Compatibility Across Model Updates
Require: Gallery indexed at step $k$; updated model at step $t$ with $\mathcal{C}^{t} \ge \mathcal{C}^{k}$; query $\mathbf{x}$; representation type repr $\in \{\mathrm{PSP}, \mathrm{LSP}\}$
Ensure: Query feature compatible with the gallery feature space at step $k$
1: Compute logits $\mathbf{z}^{t}(\mathbf{x})$
2: if repr = PSP then
3:     $\mathbf{p}^{t}(\mathbf{x}) \leftarrow \sigma(\mathbf{z}^{t}(\mathbf{x}))$
4:     if $\mathcal{C}^{t} > \mathcal{C}^{k}$ then
5:         $\tilde{\mathbf{h}}^{t\to k}(\mathbf{x}) \leftarrow \mathbf{P}_{t,k}\mathbf{p}^{t}(\mathbf{x})$
6:     else
7:         $\tilde{\mathbf{h}}^{t\to k}(\mathbf{x}) \leftarrow \mathbf{P}_{k,k}\mathbf{p}^{t}(\mathbf{x})$
8: else if repr = LSP then
9:     if $\mathcal{C}^{t} > \mathcal{C}^{k}$ then
10:        $\tilde{\mathbf{h}}^{t\to k}(\mathbf{x}) \leftarrow \mathbf{P}_{t,k}\mathbf{z}^{t}(\mathbf{x})$
11:    else
12:        $\tilde{\mathbf{h}}^{t\to k}(\mathbf{x}) \leftarrow \mathbf{z}^{t}(\mathbf{x})$
13: if $\|\tilde{\mathbf{h}}^{t\to k}(\mathbf{x})\|_{2} > 0$ then
14:     $\mathbf{h}^{t\to k}(\mathbf{x}) \leftarrow \tilde{\mathbf{h}}^{t\to k}(\mathbf{x})/\|\tilde{\mathbf{h}}^{t\to k}(\mathbf{x})\|_{2}$
15: else
16:     $\mathbf{h}^{t\to k}(\mathbf{x}) \leftarrow \mathbf{0}$
17: Compare $\mathbf{h}^{t\to k}(\mathbf{x})$ with $\{\mathbf{h}^{k}(\mathbf{g})\}_{\mathbf{g}\in\mathcal{G}}$
18: Return the ranked gallery items

Theoretical Analysis of the Backward-Compatibility Definition

Proof. Let $\mathbf{X}_{1}, \mathbf{X}_{2}$ be two independent random vectors from two vMF distributions with parameters respectively $(\boldsymbol{\mu}_{1}, \kappa_{1})$ and $(\boldsymbol{\mu}_{2}, \kappa_{2})$, where $\boldsymbol{\mu}_{1}, \boldsymbol{\mu}_{2}$ are unit vectors indicating the mean directions around which the data are concentrated and $\kappa_1$, $\kappa_2$ are non-negative scalars that determine the concentrations of the distributions around their mean (large values of $\kappa$ correspond to high concentration). By their independence, the expected cosine similarity between $\mathbf{X}_1,\mathbf{X}_2$ is given by the product of their expectations:

where the expected values of $\mathbf{X}_1$ and $\mathbf{X}_2$ are respectively [5]:

where $m_d(\kappa)$ is the expected cosine of the angle between the random vector and $\boldsymbol{\mu}$, which is expressed as:

with $I_\nu$ the modified Bessel function of the first kind of order $\nu$.

From Eq. (22), Eq. (23), and Eq. (24), we obtain:

where $\boldsymbol{\mu}_1^{\mathsf{T}}\boldsymbol{\mu}_2$ is the cosine of the angle between mean directions $\boldsymbol{\mu}_1$ and $\boldsymbol{\mu}_2$.

Since the cosine distance between $\mathbf{X}_1$ and $\mathbf{X}_2$ is $\left(1-\mathbf{X}_1^{\mathsf{T}}\mathbf{X}_2\right)$, it follows that:

which provides an analytical form that relates the expected distance between two random vectors to the parameters of the two vMF distributions.

Proof. Consider each feature of the model represented as an independent random vector. Then the compatibility inequalities of Eq. 1a and Eq. 1b in expectation can be expressed as:

where $\mathbf{X}_1,\mathbf{X}_2 \in\mathbb{R}^d$ are two independent random vectors with $\mathbb{E}\left[1-(\mathbf{X}_1)^{\mathsf{T}}\mathbf{X}_2\right]$ their expected cosine distance, and $t$ and $k$ are two distinct update steps.

Assuming that features follow a vMF distribution, according to Lemma G.1 the expected cosine distance between two independent random vectors $\mathbf{X}_1,\mathbf{X}_2 \in\mathbb{R}^d$ has the form:

where $\boldsymbol{\mu}_1,\boldsymbol{\mu}_2$ are the mean directions of the two distributions and $\kappa_1,\kappa_2$ their concentrations. Substituting Eq. (28) into Eq. 25a and Eq. 25b gives:

Then, since $m_d(\kappa)$ is a strictly increasing function of the concentration parameter $\kappa$ ([5]), and $\kappa_1^t > \kappa_1^k > 0$ across updates⁶, it follows that $m_d(\kappa_1^t)m_d(\kappa_2^t) > m_d(\kappa_1^k)m_d(\kappa_2^k)$.

It follows that:

• Eq. 27a is satisfied when the mean directions of the same class remain the same across updates, i.e., $(\mu_1^t)^\mathsf{T}\mu_2^k = 1$.

• Eq. 27b is satisfied when the angle between prototypes of different classes is greater than $\pi/2$, i.e., $(\mu_1^k)^\mathsf{T}\mu_2^k = (\mu_1^t)^\mathsf{T}\mu_2^k < 0$—note that $(\mu_1^k)^\mathsf{T}\mu_2^k = (\mu_1^t)^\mathsf{T}\mu_2^k$ due to the alignment of the prototypes required to satisfy Eq. 27a.

These two requirements are precisely conditions (a) and (b) of Theorem 3.6, which concludes the proof.

Remark 1 (Dependence on concentration). The guarantee of Theorem 3.6 is conditional on the concentration of the learned representations. From Eq. (25), for fixed prototype geometry, the separation between same-class and different-class expected cosine distances is governed by the mean resultant length of the vMF distribution, which is strictly increasing in $\kappa$ ([43]). As $\kappa$ approaches zero, the mean resultant length approaches zero and the expected cosine distances approach 1, reducing the separation between the two cases. Thus, the theorem does not impose a condition on classification accuracy; its distributional requirement is instead that the updated representation is more concentrated around its class prototypes than the old one.

Remark 2 (Role of the vMF assumption). Theorem 3.6 is stated under the vMF assumption, which is adopted as a tractable modeling assumption for class-conditional hyperspherical representations together with the explicit first-moment expression

At the expectation level, this first-moment characterization constitutes the distribution-specific quantity entering the proof. Indeed, for independent unit-norm random vectors $\mathbf{X}_1$ and $\mathbf{X}_2$,

where $d_{\mathrm{cos}}(\mathbf{x},\mathbf{y}) = 1 - \mathbf{x}^{\mathsf{T}}\mathbf{y}$.

Hence, the vMF assumption enters the expected-distance analysis through its first moment. For a class-conditional distribution on the unit hypersphere with a nonzero first moment, this quantity can be expressed as

where $\boldsymbol{\mu}$ denotes the mean direction and $\rho= \lVert\mathbb{E}[\mathbf{X}]\rVert$ denotes the mean resultant length ([60, 61]). This formulation identifies the mean direction and mean resultant length as the first-moment quantities entering the expected-distance expression. For the vMF model considered in Theorem 3.6, $\rho= m_d(\kappa)$, and the strict monotonicity of $m_d(\kappa)$ with respect to $\kappa$ ([43]) provides an explicit relation between the concentration parameter $\kappa$ and the mean resultant length appearing in the proof.

As a direct consequence of Theorem 3.6, the following corollary holds.

Proof. Let the normalized class prototypes at update step $k$ form a regular simplex centered at the origin. For any two distinct classes $i \ne j$, their pairwise inner product is

where $C^k$ denotes the number of classes defining the simplex at step $k$. Hence, the angle between any two distinct class prototypes is greater than $\pi/2$.

For the same-class case, alignment of corresponding prototypes across updates gives $(\boldsymbol{\mu}_i^t)^{\mathsf{T}}\boldsymbol{\mu}_i^k = 1$, which satisfies the geometric condition required by Eq. 27a. For two different classes $i \ne j$, prototype alignment gives

which satisfies the geometric condition required by Eq. 27b.

Therefore, all geometric conditions of Theorem 3.6 are satisfied, and, under its vMF and increasing-concentration assumptions, the compatibility inequalities of Eq. 1a and Eq. 1b hold in expectation.

Fig. 7 illustrates the expected compatibility inequalities characterized above for hyperspherical regular-simplex representations across different representation dimensions. The two update steps are denoted by $k$, corresponding to the old model, and $t$, corresponding to the updated model, whose class-conditional vMF distributions have higher concentration. In the same-class case, Figure 7a shows that

consistent with Eq. 1a. Conversely, for different classes, Figure 7b shows that

consistent with Eq. 1b.

Empirical Evidence of Increased Concentration Across Model Updates

In this appendix, we empirically assess the concentration-increase assumption discussed in § 3.4, showing that the distributions of both logits and softmax outputs become more compact around their respective class prototypes as model updates progress. To quantify the compactness, we use the mean cosine distance between the feature representations and their respective class prototypes. Specifically, let $\mathcal{Z}_c = \{z_i^c\}_{i=1}^{N_c}$ denote the set of extracted logits for class $c$, where $N_c$ represents the number of samples belonging to class $c$ in the dataset. The class prototype is defined as the mean of the $\ell_2$-normalized extracted logits, $\bar{z}_c = \frac{1}{N_c} \sum_{i=1}^{N_c} \frac{z_i^c}{\lVert z_i^c \rVert}$, and the mean cosine distance between each feature and the prototype of class $c$ is given by

The same quantities are computed for the softmax outputs by replacing $z_i^c$ with $\sigma(z_i^c)$. Lower values of $\delta_c$ correspond to larger empirical mean resultant lengths (Remark 2) and therefore indicate a tighter concentration of the normalized representations around their class mean directions.

We conduct experiments on the CIFAR-10 dataset ([50]) under two scenarios analogous to those in § 4.

Extended Classes Scenario. In this scenario, we investigate the concentration of the representations when the number of classes is expanded during a model update. The old model is trained on the first 5 classes of CIFAR-10, and the updated model is trained on the complete set of 10 classes. Both models use a ResNet-18 architecture ([39]) trained for 100 epochs using stochastic gradient descent (SGD) with an initial learning rate of 0.1, momentum of 0.9, and a cosine annealing learning rate schedule. The results shown in Figure 8 demonstrate that both the logits and the softmax outputs of the updated model are more concentrated than those of the old model for all 10 classes. This is consistent with the semi-open-set scenario, where the new classes are unknown to the old model but are known to the updated model, thus explaining the significant gap in concentration in the last 5 classes. The increase in concentration also holds for the 5 shared classes, on which both models are already highly concentrated, supporting the assumption of increasing concentration with class expansion during model updates.

Independently Trained Models in Public Repositories Scenario. We compare two models pretrained on ImageNet-1K and obtained from torchvision ([59]): ResNet-18 as the old model, and ResNet-50 as the updated model. We evaluate the concentration metric $\delta_c$ on the CIFAR-10 test set, with images resized to 224 $\times$ 224 by bicubic interpolation. As shown in Figure 8, the updated model yields more concentrated representations than the old model, even in an open-set scenario where the evaluation distribution differs from the training one: the logits are more concentrated for 7 of the 10 classes, and the softmax outputs for all 10 classes.

The observed overall increase in logit concentration is consistent with the collapse behavior discussed in § 3.2.2, and the softmax outputs exhibit the same trend. In the ideal Neural Collapse limit, the two representations behave identically in this respect: the logits of each class converge to a common vector, and so do the corresponding softmax outputs [70].

Since the hyperspherical simplex representations in Eq. (6) and Eq. (5) are constructed from the logits and softmax outputs, respectively, these results directly support assumption (ii) for both LSP and PSP. Overall, the results provide empirical evidence of increasing first-moment concentration across model updates. Under the vMF modeling assumption adopted in Theorem 3.6, this behavior is consistent with an increase in the corresponding concentration parameter $\kappa$. At the same time, the few classes whose logits become less concentrated under domain shift show that the concentration-increase assumption holds predominantly but need not hold for every class.

Empirical Analysis of the Centering Projection and the Angular Correction

In this appendix, we empirically examine the contribution of the two operations on which the design of PSP relies: the centering projection $\mathbf{P}_{k,k}$ and the deterministic transformation $\mathbf{P}_{t,k}$ for angular correction, motivated by condition (b) and condition (a) of Theorem 3.6, respectively (§ 3.2.1 and § 3.3).

Experimental setting. We consider an extended-classes update on CIFAR-10 [50] with a ResNet-18 backbone [39]; the old model is trained on the first 5 classes and the updated model on all 10 classes. All models are trained from scratch for 60 epochs using SGD with an initial learning rate of 0.1, momentum of 0.9, weight decay of $5 \cdot10^{-4}$, batch size of 128, and cosine annealing. Each model is trained with 3 different seeds, and all $3 \times3$ combinations of old and updated models are evaluated, yielding a total of 9 model pairs. The old models reach a test accuracy of 95.10–95.70% on their 5 classes, and the updated models 94.57–94.97% on all 10 classes.

The inequalities of Def. 3.1 are evaluated with the cosine distance $d$ on the 5,000 test images of the 5 shared classes, over all ordered pairs $(i,j)$ with $i \ne j$, which form the set $\mathcal{S}$ of 4,995,000 same-class pairs and the set $\mathcal{D}$ of 20,000,000 different-class pairs per model pair. For each inequality, we compute the average margin

where $\widehat{\mathbb{E}}_{\mathcal{P}}$ denotes the empirical average over the pairs in $\mathcal{P}$. A positive margin indicates that the corresponding inequality is satisfied on average over the evaluated pairs, providing an empirical counterpart to the expectation-level inequalities of Theorem 3.6.

We compare four configurations, which differ only in the operations applied to the softmax outputs before $\ell_2$-normalization. In all of them, the gallery features $\mathbf{h}^k$ are extracted with the old model and the query features $\mathbf{h}^t$ with the updated model; when their dimensions differ, $\mathbf{h}^k$ is zero-padded to $C^t$ dimensions. The configurations are defined as follows:

• $\ell_2$-Softmax: directly $\ell_2$-normalized softmax outputs for both models, without either operation.

• PSP w/o $\mathbf{P}_{t,k}$: both models use the representation of Eq. (5), each centered with $\mathbf{P}_{k,k}$ and $\mathbf{P}_{t,t}$, respectively, without angular correction.

• PSP w/o $\mathbf{P}_{k,k}$: $\ell_2$-normalized softmax outputs for the gallery, and queries mapped by $[\mathbf{I}_{C_k}\mid0]$, i.e., $\mathbf{P}_{t,k}$ without centering, which only discards the coordinates of the new classes.

• PSP: the proposed hyperspherical representation of Eq. (5), with queries mapped by $\mathbf{P}_{t,k}$.

Method	$\mathbf{P}_{k,k}$	$\mathbf{P}_{t,k}$	$\gamma_{\mathrm{same}}$	$\gamma_{\mathrm{diff}}$	Compatible pairs
$\ell_2$-Softmax	–	–	$-0.0111 \pm0.0030$	$-0.0055 \pm0.0009$	0/9
PSP w/o $\mathbf{P}_{t,k}$	✓	–	$-0.0568 \pm0.0033$	$-0.0142 \pm0.0008$	0/9
PSP w/o $\mathbf{P}_{k,k}$	–	✓*	$+0.0120 \pm0.0019$	$-0.0133 \pm0.0010$	0/9
PSP	✓	✓	$+0.0155 \pm0.0025$	$+0.0039 \pm0.0006$	9/9

In PSP w/o $\mathbf{P}_{k,k}$, i.e., without centering, restricting the queries with $[\mathbf{I}_{C_k}\mid0]$ makes $\gamma_{\mathrm{same}}$ positive in all model pairs, as it discards the probability mass that the updated model assigns to the new classes, which has no counterpart in the gallery features. However, $\gamma_{\mathrm{diff}}$ remains negative in all model pairs for both $\ell_2$-Softmax and PSP w/o $\mathbf{P}_{k,k}$, as expected from the analysis of § 3.2.1: since the softmax outputs lie in the non-negative orthant, the angle between prototypes of different classes is at most $\pi/2$ and condition (b) of Theorem 3.6 is not satisfied. In this case, the inner products between features of different classes are non-negative, so that an increase in concentration cannot yield a positive different-class margin. The centering projection $\mathbf{P}_{k,k}$ removes this limitation by mapping the prototypes onto a regular simplex centered at the origin, which, together with the alignment restored by $\mathbf{P}_{t,k}$, accounts for the positive margins of PSP.

Overall, the two operations play complementary roles: $\mathbf{P}_{k,k}$ establishes angles greater than $\pi/2$ between prototypes of different classes (condition (b)), while $\mathbf{P}_{t,k}$ restores the alignment of corresponding prototypes that this centered geometry loses under class expansion (condition (a)). Among the evaluated configurations, only their combination, i.e., PSP, yields positive average margins for both inequalities of Def. 3.1 in all model pairs, providing empirical support for the geometric conditions identified by Theorem 3.6.

Implementation Details For Standard Compatibility Benchmarks

In the following, we report the implementation details used in the experiments of § 4.2.

Experiments on CIFAR-100 ([50]). For this set of experiments, the image size is $32 \times32$. Both ResNet-18 and ResNet-50 architectures are used with the following hyperparameters for training: number of epochs 120; batch size 128; SGD optimizer with an initial learning rate of 0.1, reduced by a factor of 10 at epochs 80 and 100; SGD momentum 0.9; weight decay $5 \cdot10^{-4}$. A temperature factor of 12 is used in the cross-entropy loss to scale logit vectors during training. Training images are subjected to random cropping, horizontal flipping, and tensor normalization. For the Extended Classes benchmark, CIFAR-100 training set is partitioned into $T$ class-disjoint subsets, each containing $100/T$ classes, where $T$ is the number of update steps.

Experiments on ImageNet-1K ([85]). For this set of experiments, the image size is $224 \times224$. Both ResNet-18 and ResNet-50 architectures are trained with the following hyperparameters: number of epochs 90; batch size 1536; SGD optimizer with an initial learning rate of 0.1, reduced by a factor of 10 at epochs 30 and 60; SGD momentum 0.9; weight decay $1 \cdot10^{-4}$. Training images are subjected to random cropping, horizontal flipping, color jitter, and tensor normalization, as is standard for ImageNet-1K training. For the Extended Classes benchmark, the ImageNet-1K training set is partitioned into $T$ class-disjoint subsets, each containing $1000/T$ classes, where $T$ is the number of update steps.

Experiments on Google Landmarks v2 ([102]). For this set of experiments, the image size is $224 \times224$. A ResNet-18 architecture, initialized from ImageNet-1K pretrained weights at every update step, is trained with the following hyperparameters: number of epochs 30; batch size 512; SGD optimizer with an initial learning rate of 0.1, reduced by a factor of 10 at epochs 5, 10, and 20; SGD momentum 0.9; weight decay $5 \cdot10^{-4}$. Training images are subjected to random cropping and tensor normalization. For the Extended Classes benchmark, the first update step uses a subset of 24,393 classes, and the remaining training set is partitioned into $T - 1$ class-disjoint subsets, each containing $(81,313 - 24,393)/(T - 1)$ classes, where $T$ is the number of update steps.

Compatibility Matrices for 5-Step Update Scenario in the Extended Classes Benchmark

In Figure 9, we present the compatibility matrices showing the CMC@1 retrieval metric values at each model update step for the 5-step scenario on the CIFAR-100 dataset of Table 1.

The matrices show that PSP and LSP achieve the highest compatibility while maintaining competitive retrieval performance across the update sequence.

Additional Results on Independently Trained Models from Public Repositories

In this appendix, we present the compatibility matrices corresponding to the experimental results reported in Table 5 of § 4.3.

Specifically, Figure 10 shows the CMC@1 values of the compatibility matrices in the open-set scenario using Places365 as the evaluation set. The diagonal entries represent same-model accuracy, while the lower triangular entries show cross-model accuracy. These experiments involve sequential model updates using AlexNet, ResNet-50, RegNetX_3.2GF, ResNet-152, and MaxViT_T, each independently pretrained on ImageNet-1K, with Places365 serving as the test set.

Furthermore, Figure 11 and Figure 12 show the compatibility matrices for closed-set and open-set scenarios, respectively, using three Vision Transformer (ViT) architectures: ViT-B/32 and ViT-B/16, both trained from scratch on ImageNet-1K, and ViT-L/16, which is pretrained using weakly supervised learning on the IG-3.6B dataset and subsequently fine-tuned end-to-end on ImageNet-1K ([91]).

The results show that in the closed-set scenario, PSP achieves the highest compatibility and retrieval performance. Conversely, in open-set scenarios, while PSP maintains superior compatibility, it exhibits reduced retrieval performance. LSP achieves higher compatibility than the Baseline but lower than PSP; however, it substantially outperforms both the Baseline and PSP in cross-model retrieval performance, as evidenced by the off-diagonal values in the compatibility matrices.

Extension to Self-Supervised and Vision-Language Encoders via Linear Probing

In addition to supervised ImageNet-pretrained classifiers from torchvision ([71, 59]), we extend our analysis to self-supervised pretrained encoders and pretrained vision-language models in a linear-probe setting. Specifically, we consider two families of publicly available pretrained models: (i) three checkpoints from the DINO family, namely DINO ([20]), DINOv2 ([68]), and DINOv3 ([89]), obtained from the Hugging Face Hub ([104]); and (ii) three CLIP variants pretrained by OpenAI, namely ViT-B/32, ViT-B/16, and ViT-L/14, obtained from the OpenCLIP repository ([45]). In all cases, the image encoder is used as a frozen feature extractor.

For each checkpoint, we train a bias-free linear classifier on top of the extracted features using ImageNet-1K as the training set for 30 epochs, while keeping the encoder weights frozen. We then construct PSP and LSP representations from the resulting classifier outputs, using softmax outputs and logits, respectively. As a baseline, we directly use the $\ell_2$-normalized encoder output without any additional training. Retrieval performance is evaluated on the ImageNet-1K test set, following the same protocol adopted in the main benchmark.

Table 7 summarizes the compatibility results obtained in this linear-probe setting on ImageNet-1K. For both DINO and CLIP model families, PSP and LSP achieve substantially higher compatibility metrics than the baseline, confirming that the proposed framework can be extended beyond standard supervised classifiers.

Method	$AC$	$AA$	$ACA$
DINO family: DINO-ViT-S $\to$ DINOv2-ViT-S/14 $\to$ DINOv3-ViT-L/16
Baseline	0	35.50	0
PSP	1.00	71.00	72.00
LSP	0.67	68.70	47.02
CLIP family: CLIP ViT-B/32 $\to$ CLIP ViT-B/16 $\to$ CLIP ViT-L/14
Baseline	0	30.70	0
PSP	1.00	60.10	59.20
LSP	0.33	52.10	16.42

For the DINO family, the observed trend is consistent with that reported for supervised ImageNet-pretrained models in § 4.3. In particular, PSP achieves the strongest compatibility results, with $AC = 1.0$ and $ACA = 72.00$, while LSP achieves $AC = 0.67$ and $ACA = 47.02$. As shown in Figure 13, the baseline achieves reasonable same-model retrieval performance, but cross-model retrieval remains negligible, indicating that independently pretrained self-supervised encoders do not naturally produce aligned representation spaces across updates.

For the CLIP family, we observe the same overall pattern, as shown in Figure 14. PSP again achieves the strongest compatibility results ($AC = 1.0$, $ACA = 59.20$), while LSP improves substantially over the baseline but remains below PSP. The consistency of this behavior across both model families provides additional empirical support for the theoretical analysis in § 3.4.

Overall, these results show that a simple linear-probe protocol provides a practical way to apply PSP and LSP to pretrained encoders that do not natively expose a classification head, including both self-supervised and vision-language models.