betaFeedback

WorksComputer Science

AlignedHPO: Efficient Surrogate-Guided Loss Hyperparameter Optimization

Abstract

Modern deep learning losses involve multiple hyperparameters, such as weights, scales, and margins, typically tuned manually or through expensive black-box optimization despite their differentiability. We propose AlignedHPO, an efficient framework for tuning differentiable loss hyperparameters using gradient alignment and surrogate validation objectives. Our method employs Jacobian-vector products to reduce hypergradient computation time and memory, while differentiable metric surrogates guide optimization toward task-specific evaluation criteria. We address gradient magnitude ambiguity and stochastic instability through normalized alignment and gradient smoothing. Experiments with generalized Focal Loss and ArcFace demonstrate effective adaptation across long-tailed classification and open-set face recognition. AlignedHPO achieves competitive or improved performance compared to implicit differentiation, with nearly 12x faster hypergradient computation. These results highlight the potential of efficient surrogate-guided loss hyperparameter optimization.

Introduction

Loss function design plays a central role in deep learning across applications such as computer vision, natural language processing, and reinforcement learning. Modern loss functions often involve multiple hyperparameters, including weighting coefficients, scales, margins, and focusing factors, which influence training dynamics and final model performance. Despite their importance, these hyperparameters are typically selected through manual experimentation or expensive black-box optimization, often requiring numerous training runs.

Beyond computational cost, hyperparameter tuning complicates the development and evaluation of new training objectives. Performance improvements may depend as much on hyperparameter selection as on the loss formulation itself, while competing methods are rarely tuned with equal care. Consequently, suboptimal hyperparameter choices can bias experimental comparisons and undermine the reliability of empirical conclusions.

Many loss hyperparameters, however, are differentiable, enabling gradient-based optimization through their influence on the training trajectory. This principle underlies hypergradient and bilevel optimization methods, which have been applied to optimization parameters, loss weights, and learned training objectives. Nevertheless, efficiently tuning general loss hyperparameters toward task-specific evaluation metrics remains challenging.

Existing approaches to automatic loss optimization typically rely on differentiation through the training trajectory or implicit differentiation. The former can introduce substantial computational and memory overhead when unrolling multiple optimization steps. The latter avoids unrolling but generally assumes an approximately stationary training objective and requires expensive inverse-Hessian–vector product approximations. These limitations make both approaches costly for online hyperparameter optimization in large neural networks.

Another challenge lies in the choice of validation objective. Evaluation metrics, such as classification accuracy and ROC AUC, are often non-differentiable, while conventional validation losses may not accurately reflect the target metric. Differentiable metric surrogates offer a way to bridge this gap, allowing loss hyperparameters to be optimized toward the desired evaluation criterion.

In this work, we propose AlignedHPO, an efficient framework for surrogate-guided optimization of differentiable loss hyperparameters during training. Our method combines local gradient alignment with forward-mode automatic differentiation, using Jacobian–vector products to efficiently estimate hypergradients without unrolling training trajectories or approximating inverse Hessians. We further address limitations of direct gradient alignment, including undesirable gradient scaling and stochastic instability, through normalized similarity objectives and historical gradient smoothing.

We evaluate AlignedHPO on generalized Focal Loss and ArcFace objectives, jointly optimizing up to hundreds of hyperparameters. Experiments on long-tailed classification demonstrate improvements over fixed loss configurations and competitive performance with implicit differentiation at substantially lower computational cost. Open-set face recognition experiments further demonstrate dataset-specific adaptation while highlighting the challenges of generalization across different validation distributions.

Our main contributions are summarized as follows:

  1. We introduce a framework for online optimization of differentiable loss hyperparameters, including nonlinear parameterizations, guided by surrogate validation objectives that approximate task-specific evaluation metrics.

  1. We develop an efficient hypergradient computation procedure based on Jacobian–vector products, substantially reducing runtime and memory consumption compared to conventional reverse-over-reverse and implicit differentiation.

  1. We analyze challenges of gradient alignment, including sensitivity to gradient magnitude and stochastic instability, and introduce normalized alignment and gradient smoothing to improve optimization robustness.

  1. We demonstrate effective loss hyperparameter adaptation on long-tailed classification and open-set face recognition, investigating both computational efficiency and generalization across evaluation distributions.

Related Work

Gradient-based hyperparameter optimization. Gradient-based HPO exploits the dependence of training on hyperparameters. Early methods differentiated through complete trajectories using reversible learning [9], followed by forward- and reverse-mode differentiation [8] and truncated backpropagation [24]. Implicit differentiation instead estimates hypergradients at approximately stationary solutions, requiring inverse-Hessian–vector product approximations [18]. Recent stochastic formulations relax stationarity assumptions [13]. However, trajectory differentiation incurs substantial computational and memory costs, while implicit methods require expensive Hessian-related computations. We focus on efficient online optimization without long unrolling or stationarity assumptions.

Adaptive weighting and auxiliary learning. Related methods adapt the contributions of training examples, tasks, and auxiliary losses. Gradient similarity prevents harmful auxiliary updates [7], while OL-AUX estimates auxiliary gradient contributions [16]. Validation-based meta-gradients enable example reweighting [22], extended to nonlinear weighting by Meta-Weight-Net [25]. Auto-λ\lambda learns task weights through bilevel optimization [17], while AMAL adapts instance-level loss mixtures [26]. Gradient-based loss balancing has also been explored in Grad-Norm [4], while PCGrad [30] uses gradient projections to mitigate conflicting updates. These approaches primarily optimize example or task weights rather than internal loss hyperparameters.

Automatic loss design. L2T-DLF learns training losses using a teacher network and trajectory differentiation [27]. AuxiLearn uses implicit differentiation to learn nonlinear combinations of auxiliary losses and tasks [21], while AutoBalance optimizes class- and group-dependent loss parameters for imbalanced classification [14]. These methods support nonlinear objectives but incur costly trajectory differentiation or iterative inverse-Hessian approximations. In contrast, we optimize general differentiable loss hyperparameters without unrolling or Hessian estimation, demonstrating our method on Focal Loss and ArcFace [6, 15].

Efficient gradient alignment and hypergradients. Reverse-mode differentiation computes one-step hypergradients without explicitly constructing mixed-derivative matrices [8, 22]. More closely related, GraP [11] optimizes linear weights of composite pretraining losses through gradient alignment in the shared embedding space, using normalization to address gradient-scale ambiguity. Our method extends alignment to nonlinear loss hyperparameters and employs forward-over-reverse Jacobian–vector

KK: auxiliary losses; TT: unrolled steps; QQ: inverse-Hessian iterations. FP/BP/JVP/HVP: forward/backward, Jacobian-vector, and Hessian-vector products. Target objective denotes the criterion guiding loss-parameter updates, not the evaluation metric. *Nonlinear combinations of predefined auxiliary losses. †Class-specific parameters of a cross-entropy-based loss. Costs exclude standard training and validation-gradient computation; constant factors and loss-level operations are omitted.

products for efficient computation. We additionally introduce historical gradient smoothing to improve stability under stochastic training.

Surrogate validation objectives. Differentiable surrogates enable optimization toward non-differentiable evaluation metrics. Pairwise losses, including sigmoid approximations, have been studied for AUC maximization [28], while EXACT [12] targets classification accuracy. We use AUC surrogates to guide online adaptation of training loss hyperparameters toward task-specific evaluation criteria.

Table 1 summarizes existing approaches by application, loss parameterization, optimization strategy, and computational complexity.

MethodApplicationNonlinear param.TuningTarget objectiveComplexity
OL-AUX [16]AUX–Gradient alignmentMain lossO(K)O(K) BP
Auto-λ\lambda [17]MTL–Bilevel (FD)Val. task lossO(1)O(1) FP/BP
AMAL [26]AUX–One-step bilevelVal. task lossO(1)O(1) BP
Meta-Weight-Net [25]Design/HPO–One-step bilevelCross-Ent.O(1)O(1) BP
L2T-DLF [27]Design/HPO✓Unrolled bilevelExpected metricO(T)O(T) HVP + mixed
ML3 [1]Design/HPO✓Unrolled bilevelMeta-task lossO(T)O(T) BP/HVP
AuxiLearn [21]AUX✓*Implicit diff.Val. main lossO(Q)O(Q) HVP
AutoBalance [14]Design/HPO✓†Implicit diff.Balanced lossO(Q)O(Q) HVP
AlignedHPODesign/HPO✓Gradient alignmentVal. surrogateO(1)O(1) JVP

Table 1. Comparison of differentiable loss optimization methods. We summarize their applications, support for nonlinear loss parameters, tuning strategies, target objectives, and computational complexity. Complexities indicate the number of differentiation operations rather than directly comparable runtimes.

Method

We consider the optimization of a training objective Ltr(θ,w)\mathcal{L}_{\mathrm{tr}}(\theta,w), where θ∈Rd\theta\in\mathbb{R}^{d} denotes the model parameters and w∈Rmw \in\mathbb{R}^{m} is a vector of differentiable loss hyperparameters. Our goal is to adapt ww during training such that updates of θ\theta improve a validation objective Lval(θ)\mathcal{L}_{\mathrm{val}}(\theta). Unlike conventional hyperparameter optimization, which evaluates multiple training trajectories, our approach exploits the local dependence of the training gradient on ww to obtain efficient hyperparameter updates.

Core Method

Consider a training step at the current model parameters θ\theta and hyperparameters w0w_0. We first approximate the training objective by its first-order Taylor expansion with respect to the hyperparameters:

Ltr(θ,w0+s)≈Ltr(θ,w0)+sT∇wLtr(θ,w0),(1)\mathcal{L}_{\mathrm{tr}}(\theta,w_0+s) \approx\mathcal{L}_{\mathrm{tr}}(\theta,w_0) + s^{\mathsf{T}}\nabla_w\mathcal{L}_{\mathrm{tr}}(\theta,w_0), \tag*{(1)}

where s=w−w0s=w-w_0 denotes a local hyperparameter perturbation.

Differentiating Eq. (1) with respect to the model parameters yields

g(s)≈g0+Gs,(2)g(s) \approx g_0+Gs, \tag*{(2)}

where

g0=∇θLtr(θ,w0),(3)g_0=\nabla_\theta\mathcal{L}_{\mathrm{tr}}(\theta,w_0), \tag*{(3)}
G=∇θ∇wLtr(θ,w0)∈Rd×m.G=\nabla_\theta\nabla_w\mathcal{L}_{\mathrm{tr}}(\theta,w_0)\in\mathbb{R}^{d\times m}.

Thus, the columns of GG describe how each loss hyperparameter modifies the training gradient. Importantly, Eq. (2) provides a linear approximation to the gradient even when the original training objective depends nonlinearly on its hyperparameters.

We now consider a single SGD update,

θ+(s)=θ−ηθg(s),(4)\theta^{+}(s)=\theta-\eta_\theta g(s), \tag*{(4)}

where ηθ\eta_\theta is the model learning rate. Let v=∇θLval(θ)v=\nabla_\theta\mathcal{L}_{\mathrm{val}}(\theta) denote the validation gradient. A first-order approximation of the validation loss after this update gives

Lval(θ+(s))≈Lval(θ)−ηθvTg(s)≈C−ηθvTGs,(5)\begin{aligned} \mathcal{L}_{\mathrm{val}}(\theta^{+}(s))\approx\mathcal{L}_{\mathrm{val}}(\theta)-\eta_\theta v^{\mathsf{T}}g(s) \\ &\approx C-\eta_\theta v^{\mathsf{T}}Gs, \tag*{(5)} \end{aligned}

where CC is independent of ss. Consequently, the local hypergradient is

∇sLval≈−ηθGTv.(6)\nabla_s\mathcal{L}_{\mathrm{val}}\approx-\eta_\theta G^{\mathsf{T}}v. \tag*{(6)}

This leads to a simple hyperparameter update,

w+=w0+ηwGTv,(7)w^{+}=w_0+\eta_wG^{\mathsf{T}}v, \tag*{(7)}

where ηw\eta_w absorbs the model learning-rate factor. The update increases the contribution of hyperparameters whose induced gradient changes are positively aligned with the validation gradient.

A direct evaluation of Eq. (6) would require computing the mixed-derivative matrix GG, which can be prohibitively large. Fortunately, the method only requires its vector–Jacobian product GTvG^{\mathsf{T}}v. By the symmetry of mixed derivatives,

GTv=Dθ[∇wLtr(θ,w0)][v],(8)G^{\mathsf{T}}v = D_{\theta}\left[\nabla_{w}\mathcal{L}_{\mathrm{tr}}(\theta,w_{0})\right][v], \tag*{(8)}

where Dθ[⋅][v]D_{\theta}[\cdot][v] denotes a directional derivative along vv.

Equation (8) can be computed using a Jacobian–vector product (JVP), without explicitly constructing GG. Since forward-mode differentiation propagates a single tangent direction through the computation, its cost does not scale linearly with the number of loss hyperparameters. In practice, this amounts to one additional forward-mode differentiation pass through the relevant computation graph, enabling efficient online hyperparameter optimization even for high-dimensional ww.

Gradient Similarity

The update in Eq. (7) maximizes the inner product between the training and validation gradients. Although this improves their local alignment, the inner product depends on both gradient direction and magnitude. Consequently, hyperparameter optimization may favor increasing the training gradient norm rather than improving its direction.

This distinction is important because the magnitude of the training gradient directly affects the effective model update size. Even with a fixed learning-rate schedule, hyperparameter-dependent changes in the gradient norm implicitly modify the effective learning rate, potentially interfering with the intended optimization dynamics.

Since our primary objective is to identify a training direction that improves validation performance, we also consider cosine similarity:

S(s)=vT(g0+Gs)∥v∥2∥g0+Gs∥2.(9)S(s) = \frac{v^{\mathsf{T}}(g_{0}+Gs)}{\lVert v\rVert_{2}\lVert g_{0}+Gs\rVert_{2}}. \tag*{(9)}

Unlike the dot product, cosine similarity is invariant to positive rescaling of the training gradient and therefore does not directly reward larger gradient magnitudes.

Differentiating Eq. (9) with respect to ss gives

∇sS(s)=GT[v∥v∥2∥g(s)∥2−vTg(s)∥v∥2∥g(s)∥23g(s)].(10)\nabla_{s}S(s) = G^{\mathsf{T}}\left[\frac{v}{\lVert v\rVert_{2}\lVert g(s)\rVert_{2}}-\frac{v^{\mathsf{T}}g(s)}{\lVert v\rVert_{2}\lVert g(s)\rVert_{2}^{3}}g(s)\right]. \tag*{(10)}

In particular, at s=0s=0,

∇sS(0)=GT∥v∥2∥g0∥2[v−vTg0∥g0∥22g0].(11)\nabla_{s}S(0)=\frac{G^{\mathsf{T}}}{\lVert v\rVert_{2}\lVert g_{0}\rVert_{2}}\left[v-\frac{v^{\mathsf{T}}g_{0}}{\lVert g_{0}\rVert_{2}^{2}}g_{0}\right]. \tag*{(11)}

The expression in brackets is the component of the validation gradient orthogonal to the current training gradient. Thus, cosine-based optimization modifies the training direction without explicitly encouraging changes in its norm. Both terms in Eq. (11) are matrix–vector products of the form GTuG^{\mathsf{T}}u and can therefore be evaluated using the JVP formulation in Eq. (8).

Gradient noise and orthogonalization. Despite its desirable scale invariance, cosine-based optimization introduces an additional challenge in stochastic training. The validation gradient and the training gradient are estimated from different data distributions and typically exhibit weaker alignment than the training gradient and its own local hyperparameter derivatives.

In particular, the correlation between vv and g0g_{0} can be small and highly variable across iterations, while columns of GG may remain strongly correlated with g0g_{0}. As a result, the orthogonalization term in Eq. (11) can dominate the hyperparameter update.

This may produce an undesirable unlearning effect: hyperparameters are adjusted primarily to suppress components of the current training gradient, rather than to increase alignment with the validation direction. Because stochastic gradients contain substantial batch-specific variation, such adjustments may reflect transient gradient fluctuations instead of systematic differences between training and validation objectives.

Historical gradient smoothing. To mitigate this effect, we replace the instantaneous gradient used for orthogonalization with its exponential moving average (EMA):

gˉt=βgˉt−1+(1−β)gt,(12)\bar{g}_{t}=\beta\bar{g}_{t-1}+(1-\beta)g_{t}, \tag*{(12)}

where β∈[0,1)\beta\in[0,1) controls the smoothing strength and gtg_{t} denotes the current training gradient.

The resulting surrogate similarity direction is defined as

ut=1∥vt∥2∥gˉt∥2[vt−vtTgˉt∥gˉt∥22gˉt].(13)u_{t}=\frac{1}{\lVert v_{t}\rVert_{2}\lVert\bar{g}_{t}\rVert_{2}}\left[v_{t}-\frac{v_{t}^{\mathsf{T}}\bar{g}_{t}}{\lVert\bar{g}_{t}\rVert_{2}^{2}}\bar{g}_{t}\right]. \tag*{(13)}

We then compute the hyperparameter update as

wt+1=wt+ηwGtTut.(14)w_{t+1}=w_{t}+\eta_{w}G_{t}^{\mathsf{T}}u_{t}. \tag*{(14)}

Here, the historical gradient is treated as a fixed reference direction when constructing utu_{t}. Thus, Eq. (14) is a surrogate hypergradient rather than the exact derivative of the instantaneous cosine. The EMA reference suppresses batch-specific fluctuations and encourages hyperparameter updates relative to the persistent training direction, instead of the potentially noisy current gradient.

Optionally, the validation gradient can also be smoothed using an EMA to reduce variance in its estimated direction. The computational advantage of the method is preserved, since the final update still requires only a single product Gt⊤utG_t^\top u_t.

We investigate both dot-product and cosine-based objectives experimentally, analyzing their effects on optimization stability and downstream performance.

Validation Loss

The proposed method relies on the validation gradient vv to guide training. However, evaluation metrics are often non-differentiable, making them unsuitable for direct gradient-based optimization. Moreover, a conventional validation loss does not necessarily provide a locally accurate representation of the target evaluation metric.

We therefore propose to construct differentiable surrogate validation objectives that directly approximate the metric of interest. Unlike training losses, which may prioritize convenient optimization properties, these surrogates are chosen to provide informative local gradient directions for improving the evaluation metric.

For classification and ranking tasks, we consider the area under the receiver operating characteristic curve (ROC AUC). Given validation scores ai+a_i^+ for positive examples and aj−a_j^- for negative examples, empirical AUC can be expressed, ignoring ties, as

AUC=1N+N−∑i=1N+∑j=1N−I[ai+>aj−](15)\mathrm{AUC} = \frac{1}{N_+N_-}\sum_{i=1}^{N_+}\sum_{j=1}^{N_-}\mathbb{I}\left[a_i^+ > a_j^-\right] \tag*{(15)}

where N+N_+ and N−N_- denote the numbers of positive and negative examples. For multiclass classification, we compute the surrogate independently for each class in a one-vs-rest setting and average the resulting losses across classes.

To obtain a differentiable objective, we replace the pairwise indicator by a sigmoid approximation:

LvalAUC=1N+N−∑i=1N+∑j=1N−σ(aj−−ai+τ)(16)\mathcal{L}_{\mathrm{val}}^{\mathrm{AUC}} = \frac{1}{N_+N_-}\sum_{i=1}^{N_+}\sum_{j=1}^{N_-}\sigma\left(\frac{a_j^- - a_i^+}{\tau}\right) \tag*{(16)}

where σ(z)=(1+e−z)−1\sigma(z) = (1 + e^{-z})^{-1} and τ>0\tau> 0 controls the smoothness of the approximation.

As τ→0\tau\to0, the surrogate approaches the pairwise misranking rate for examples with unequal scores, corresponding to 1−AUC1-\mathrm{AUC}. For finite τ\tau, it provides nonzero gradients around the decision boundary, allowing the method to identify hyperparameter changes that improve pairwise ranking.

The validation direction is then obtained as

v=∇θLvalAUC(θ).(17)v = \nabla_\theta\mathcal{L}_{\mathrm{val}}^{\mathrm{AUC}}(\theta). \tag*{(17)}

By constructing the validation objective around the actual evaluation criterion, we aim to improve the local relevance of the hyperparameter updates. This distinction is particularly important when the training objective and the target metric emphasize different properties of the predictions.

Experiments

In this section, we evaluate the computational efficiency of the proposed method and its effectiveness in tuning loss hyperparameters across tasks of increasing complexity. We consider a controlled toy example, closed-set classification with shared training and validation classes, and open-set face recognition, where distribution differences introduce additional instability in gradient alignment.

Computational Efficiency

We compare the computational efficiency of the proposed Jacobian–vector-product (JVP) approach against two alternative hypergradient computation methods: reverse-over-reverse automatic differentiation and implicit differentiation based on a truncated Neumann series, as employed in AuxiLearn [21] and AutoBalance [14]. Experiments use a ResNet-50 backbone with ArcFace loss, a batch size of 32, and 1,000 class-specific margin hyperparameters on an NVIDIA RTX 4060 GPU. Implicit differentiation uses five Neumann iterations. All methods share a precomputed validation gradient, excluded from the measurements. We report median computation time and incremental peak GPU memory consumption.

As shown in Figure 1a, the proposed JVP approach requires 81.4 ms, compared to 162.3 ms for reverse-over-reverse and 974.4 ms for implicit differentiation, achieving speedups of 1.99×\times and 11.97×\times, respectively. JVP and reverse-over-reverse compute the same local hypergradient, so the improvement comes solely from the differentiation strategy.

Computational efficiency of hypergradient estimation for an ArcFace model

Figure 1. Computational efficiency of hypergradient estimation for an ArcFace model with a ResNet-50 backbone (batch size 32). We compare implicit differentiation using a five-step Neumann approximation, conventional reverse-over-reverse differentiation, and our forward-over-reverse implementation (AlignedHPO). Lower values are better.

Similarly, Figure 1b shows that JVP requires 1,703.1 MiB, compared to 2,061.2 MiB and 3,055.9 MiB, reducing memory consumption by 17.4% and 44.3%, respectively.

The higher cost of implicit differentiation results primarily from repeated Hessian–vector products used to approximate the inverse Hessian. Unlike the local methods, it estimates the sensitivity of an approximately stationary inner solution and thus computes a different hypergradient approximation.

Overall, forward-over-reverse differentiation substantially reduces both runtime and memory requirements, making it well suited for online loss hyperparameter optimization.

Toy Example: Aligning Training and Validation Objectives

We consider a one-dimensional binary classification problem with overlapping class-conditional distributions, x∣y=1∼U(0,2)x \mid y = 1 \sim\mathcal{U}(0, 2) and x∣y=0∼U(0,4)x \mid y = 0 \sim\mathcal{U}(0, 4), with equal class proportions. The classifier is a linear model fθ(x)=s(x−b)f_\theta(x) = s(x-b), where ss and bb are learnable parameters.

We optimize the weighted binary cross-entropy loss,

L(θ,p)=−E[pylog⁡σ(fθ(x))+(1−p)(1−y)log⁡(1−σ(fθ(x)))](18)\mathcal{L}(\theta,p)=-\mathbb{E}\left[p y\log\sigma(f_\theta(x))+(1-p)(1-y)\log(1-\sigma(f_\theta(x)))\right] \tag*{(18)}

where pp controls the relative class weights.

We fix the validation weight to pval=0.8p_{\mathrm{val}} = 0.8 and initialize the training weight at ptrain=0.5p_{\mathrm{train}} = 0.5. We compare standard BCE (p=0.5p = 0.5), an oracle baseline matching the validation objective (p=0.8p = 0.8), and AlignedHPO, which automatically tunes pp using gradient alignment. All methods train the classifier using SGD (η=0.1\eta= 0.1); AlignedHPO additionally optimizes pp using Adam (η=0.01\eta= 0.01). Results are averaged over five runs with independently sampled training and validation sets.

Figure 2 shows the training and validation losses and the evolution of pp. AlignedHPO reduces the validation loss from approximately 0.55 to 0.20, matching the oracle baseline and outperforming standard BCE, which converges to approximately 0.26. Meanwhile, the learned training weight progressively increases from 0.5 toward 0.8, closely approaching the validation weight without explicitly targeting its value. This demonstrates that gradient alignment can automatically adapt an initially mismatched training objective to recover oracle-level validation performance.

Figure 2. Toy example of automatic loss hyperparameter adaptation. (a) Training loss evaluated with each method’s own class weight. (b) Validation loss evaluated with the shared weight pval=0.8p_{\mathrm{val}} = 0.8. (c) Evolution of the training class weight pp. AlignedHPO automatically adapts the training objective, achieving validation performance comparable to the oracle baseline with p=0.8p = 0.8. Solid lines show the mean over five independent runs; shaded regions indicate one standard deviation.

Long-Tailed Classification

We evaluate AlignedHPO on CIFAR-10-LT and CIFAR-100-LT [2, 5], constructed by exponentially reducing class frequencies in the training sets. We use an imbalance ratio of 100 between the most and least frequent classes, while validation and test sets remain balanced. This creates a mismatch between the training distribution and the target evaluation metric.

Parameterized Focal Loss. We consider a generalized Focal Loss with class-dependent weights αc\alpha_c and focusing parameters γc\gamma_c:

Lfocal=−1B∑i=1Bαyi(1−pi,yi)γyilog⁡pi,yi,(19)\mathcal{L}_{\mathrm{focal}}=-\frac{1}{B}\sum_{i=1}^{B}\alpha_{y_i}(1-p_{i,y_i})^{\gamma_{y_i}}\log p_{i,y_i}, \tag*{(19)}

where BB is the batch size and pi,yip_{i,y_i} is the predicted probability of the ground-truth class. We optimize 2C2C hyperparameters, with class weights normalized using softmax:

αc=Cexp⁡(ac)∑j=1Cexp⁡(aj).(20)\alpha_c=\frac{C\exp(a_c)}{\sum_{j=1}^{C}\exp(a_j)}. \tag*{(20)}

The focusing parameters γc\gamma_c are constrained to be non-negative.

Compared Methods. We compare standard cross-entropy and Focal Loss with fixed γ=1\gamma= 1 against three adaptive variants: implicit differentiation (Implicit Diff.) using a truncated Neumann approximation, and AlignedHPO with dot-product and cosine-based gradient alignment.

All adaptive methods optimize the same class-specific Focal Loss hyperparameters using the differentiable ROC AUC surrogate introduced in § 3.3. We use sigmoid temperatures of τ=0.01\tau= 0.01 for CIFAR-10-LT and τ=0.001\tau= 0.001 for CIFAR-100-LT. The balanced validation sets guide hyperparameter optimization, enabling a fair comparison of different hypergradient estimation strategies.

Training Details. We use a ResNet-18 [9] backbone, trained for 200 epochs using SGD with momentum 0.9 and an initial learning rate of 0.1. The learning rate is multiplied by 0.01 at epochs 160 and 180. We use a batch size of 128 for all methods except implicit differentiation, which uses 96 due to its higher memory requirements. The Focal Loss parameters are initialized with uniform class weights (αc=1\alpha_c = 1) and zero focusing parameters (γc=0\gamma_c = 0), corresponding to standard cross-entropy.

Results and Discussion. Table 2 reports test classification accuracy on both benchmarks.

MethodCIFAR10-LTCIFAR100-LT
Cross-Entropy71.3339.92
Focal (γ=1\gamma= 1)72.3939.83
Implicit Diff.74.5340.48
AlignedHPO (Dot)74.7940.65
AlignedHPO (Cosine)74.4142.00

Table 2. Classification accuracy (%) on CIFAR10-LT and CIFAR100-LT. Higher is better.

In our experiments, all adaptive methods outperform the fixed-loss baselines, demonstrating the effectiveness of surrogate-guided hyperparameter optimization. AlignedHPO improves upon standard cross-entropy by up to 3.46 and 2.08 percentage points on CIFAR-10-LT and CIFAR-100-LT, respectively. It also achieves slightly better results than implicit differentiation, while offering substantially lower computational cost (§ 4.1).

The two alignment objectives exhibit different behavior. Dot-product alignment performs slightly better on CIFAR-10-LT (74.79% vs. 74.41%), whereas cosine alignment achieves a larger improvement on CIFAR-100-LT (42.00% vs. 40.65%). This suggests that controlling gradient magnitude may be more beneficial in more challenging classification settings.

Figure 3 illustrates the evolution of learned hyperparameters on CIFAR-10-LT. We expect larger class weights αc\alpha_c for rare classes to compensate for imbalance, and larger focusing parameters γc\gamma_c for frequent classes to suppress their easier examples. The learned parameters partially follow this intuition: α0\alpha_0 decreases substantially, while γ0\gamma_0 increases, reducing the contribution of the most frequent class. Meanwhile, α5\alpha_5 increases, although the rarest class retains a weight close to its initialization. The focusing parameters for classes 5 and 9 remain close to zero.

Figure 3. Evolution of the effective Focal Loss parameters α\alpha (top) and γ\gamma (bottom) for selected classes on CIFAR-10-LT. We compare AlignedHPO, its cosine variant, and implicit differentiation.

Notably, both AlignedHPO variants produce smooth and similar hyperparameter trajectories, whereas implicit differentiation exhibits substantially larger fluctuations and occasional sharp changes. This indicates more stable hyperparameter adaptation with the proposed local gradient alignment approach, despite its considerably lower computational cost.

Open-Set Face Recognition

We further evaluate AlignedHPO on open-set face recognition, where training and evaluation identities are disjoint. Unlike closed-set classification, this introduces an additional mismatch between training and validation objectives, making hyperparameter optimization more challenging.

We train models on CASIA-WebFace [29] and evaluate verification accuracy on LFW [10], AgeDB-30 [20], and CFP-FP [23], which capture variations in imaging conditions, age, and pose, respectively.

Parameterized ArcFace Loss. We adopt ArcFace as the training objective, which introduces an additive angular margin into normalized softmax classification:

LArcFace=−1B∑i=1Blog⁡escos⁡(θyi,i+m)escos⁡(θyi,i+m)+∑j≠yiescos⁡θi,j(21)\mathcal{L}_{\mathrm{ArcFace}} = -\frac{1}{B}\sum_{i=1}^{B}\log\frac{e^{s\cos(\theta_{y_i,i}+m)}}{e^{s\cos(\theta_{y_i,i}+m)}+\sum_{j\ne y_i}e^{s\cos\theta_{i,j}}} \tag*{(21)}

where θi,j\theta_{i,j} denotes the angle between the normalized embedding and class weight, mm is the angular margin, and ss is the logit scaling factor. We treat these parameters as differentiable hyperparameters and optimize them during training.

Validation Objective. Face verification relies on similarity scores between image pairs rather than class predictions.

We compute cosine similarity between normalized embeddings and use the differentiable binary ROC AUC surrogate from § 3.3 to guide hyperparameter optimization. This allows the classification-based training loss to be adapted toward the verification objective.

Dataset-Specific Optimization. We consider three variants of AlignedHPO, independently tuned using validation pairs from LFW, AgeDB-30, and CFP-FP. All variants share the same CASIA-WebFace training data and differ only in the validation distribution used for tuning. We evaluate both dot-product and cosine-based alignment against standard ArcFace with fixed hyperparameters (s=32s = 32, m=0.5m = 0.5).

Training Details. We use a MobileFaceNet backbone [3] trained for 18 epochs with SGD, Nesterov momentum of 0.9, and weight decay of 5×10−45 \times10^{-4}. The initial learning rate is 0.01 and is multiplied by 0.1 at epochs 6, 11, and 16.

Since the ArcFace scale parameter strongly affects the training gradient magnitude, we normalize the gradients using an exponential moving average (EMA) of their norm, with a smoothing factor of 0.9. This reduces variations in the effective learning rate caused by hyperparameter adaptation and ensures more comparable optimization dynamics across methods.

Results and Discussion. Table 3 reports verification accuracy on all three benchmarks.

MethodLFWAgeDBCFP-FP
ArcFace (original)97.6886.3787.34
AlignedHPO – Dot product
AlignedHPO (LFW)97.1584.3386.63
AlignedHPO (AgeDB)96.9885.8779.44
AlignedHPO (CFP-FP)96.6882.4589.26
AlignedHPO – Cosine similarity
AlignedHPO (LFW)96.7382.4084.01
AlignedHPO (AgeDB)97.1385.4581.97
AlignedHPO (CFP-FP)96.6881.5888.17

Table 3. Face verification performance (%) on LFW, AgeDB-30, and CFP-FP. Each AlignedHPO variant is tuned using the corresponding validation dataset. The retrained variants use the parameters identified during hyperparameter optimization.

The results demonstrate dataset-specific adaptation of the learned loss hyperparameters. For both alignment objectives, tuning on CFP-FP achieves the highest CFP-FP accuracy among adaptive variants, while tuning on AgeDB-30 provides the best AgeDB-30 performance. A similar trend is observed on LFW with dot-product alignment. This indicates that AlignedHPO can adapt the training objective to the characteristics of the target validation distribution, even when training and validation identities are disjoint.

However, the original ArcFace baseline remains superior on LFW and AgeDB-30. The main improvement is observed on CFP-FP, where dot-product alignment achieves 89.26%, exceeding the baseline by 1.92 percentage points. These results suggest that optimizing hyperparameters for a single validation distribution can improve performance on that target but may reduce generalization to other verification benchmarks.

Another possible limitation is the mismatch between the validation objective and evaluation metric: we optimize a differentiable ROC AUC surrogate but evaluate verification accuracy. Although related, these metrics are not equivalent, potentially limiting the effectiveness of hyperparameter adaptation.

Future work could address these limitations by jointly optimizing hyperparameters across multiple validation datasets and exploring differentiable accuracy surrogates [12]. These extensions may improve the robustness and generalization of the learned training objective.

Limitations

AlignedHPO has several limitations. First, its local, single-step approximation neglects the long-term effects of hyperparameters on training, potentially limiting effectiveness when delayed effects are important.

Second, the method depends on validation gradient quality. Differences between training and validation distributions can introduce noise and instability. Although gradient smoothing mitigates these effects, our open-set experiments show that dataset-specific adaptation does not necessarily generalize across evaluation benchmarks.

Finally, performance depends on how well the differentiable validation objective approximates the target metric. In particular, our ROC AUC surrogate does not directly optimize verification accuracy. Future work could explore multi-dataset validation objectives, alternative metric surrogates, and extensions beyond local hypergradient estimation.

Conclusion

We introduced AlignedHPO, an efficient framework for online optimization of differentiable loss hyperparameters. By combining local gradient alignment with Jacobian–vector products, the proposed method avoids expensive trajectory differentiation and inverse-Hessian approximations while supporting nonlinear loss parameterizations.

We further investigated dot-product and cosine-based alignment and introduced gradient smoothing to improve optimization stability. Experiments demonstrate substantial reductions in hypergradient computation time and memory, along with competitive or improved performance on long-tailed classification and open-set face recognition. Our results also highlight the potential of differentiable metric surrogates for guiding loss hyperparameter optimization, providing a practical foundation for automated loss adaptation during deep network training.

LLM Usage Statement

OpenAI ChatGPT was used to assist with scientific writing and editing, related-work analysis, code development, and experimental plotting. All technical contributions, experimental results, and conclusions were developed and verified by the authors.

References

  1. [1]Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4161–4168. IEEE, 2021. 3
  2. [2]Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. 2019. 6arxiv.org/abs/1906.07413
  3. [3]Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. Mobilefacenet: Efficient cnns for accurate real-time face verification on mobile devices. In Chinese conference on biometric recognition, pages 428–438. Springer, 2018. 8arxiv.org/abs/1804.07573
  4. [4]Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794–803. PMLR, 2018. 2
  5. [5]Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 9260–9269. IEEE, 2019. 6arxiv.org/abs/1901.05555
  6. [6]Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019. 2DOI
  7. [7]Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity, 2018. 2arxiv.org/abs/1812.02224
  8. [8]Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International conference on machine learning, pages 1165–1173. PMLR, 2017. 2arxiv.org/abs/1703.01785
  9. [9]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6arxiv.org/abs/1512.03385
  10. [10]Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report, 2008. 7
  11. [11]Ivan Karpukhin and Andrey Savchenko. When losses align: Gradient-based composite loss weighting for efficient pre-training. arXiv preprint arXiv:2605.07756, 2026. 2arxiv.org/abs/2605.07756
  12. [12]Ivan Karpukhin, Stanislav Dereka, and Sergey Kolesnikov. Exact: How to train your accuracy. Pattern Recognition Letters, 185:23–30, 2024. 3, 8DOI
  13. [13]Minyoung Kim and Timothy Hospedales. A stochastic approach to bi-level optimization for hyperparameter optimization and meta learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 17913–17920, 2025. 2arxiv.org/abs/2410.10417
  14. [14]Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Autobalance: Optimized loss functions for imbalanced data. pages 3163–3177, 2021. 2, 3, 5arxiv.org/abs/2201.01212
  15. [15]Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 2
  16. [16]Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning, 2019. 2, 3
  17. [17]Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.03091, 2022. 2, 3arxiv.org/abs/2202.03091
  18. [18]Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics, pages 1540–1552. PMLR, 2020. 2arxiv.org/abs/1911.02590
  19. [20]Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pages 1997–2005. IEEE, 2017. 7DOI
  20. [21]Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. In International Conference on Learning Representations, 2021. 2, 3, 5arxiv.org/abs/2007.02693
  21. [22]Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334–4343. PMLR, 2018. 2arxiv.org/abs/1803.09050
  22. [23]Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV), pages 1–9. IEEE, 2016. 7DOI
  23. [24]Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back-propagation for bilevel optimiza- tion. In The 22nd international conference on artificial intelligence and statistics, pages 1723–1732. PMLR, 2019. 2
  24. [25]Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. 2019. 2, 3arxiv.org/abs/1902.07379
  25. [26]Durga Sivasubramanian, Ayush Maheshwari, Pradeep Shenoy, Ganesh Ramakrishnan, et al. Adaptive mixing of auxiliary losses in supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9855–9863, 2020. 2, 3
  26. [27]Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, and Tie-Yan Liu. Learning to teach with dynamic loss functions. 2018. 2, 3arxiv.org/abs/1810.12081
  27. [28]Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey. ACM computing surveys, 55 (8):1–37, 2022. 3arxiv.org/abs/2203.15046
  28. [29]Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7923, 2014. 7arxiv.org/abs/1411.7923
  29. [30]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. pages 5824–5836, 2020. 2arxiv.org/abs/2001.06782

Paper details

vlnc:W_4escckmu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

OpenAI ChatGPT

View statement
Looking for related papers…
Contents