WorksComputer Science
AlignedHPO: Efficient Surrogate-Guided Loss Hyperparameter Optimization
Abstract
Modern deep learning losses involve multiple hyperparameters, such as weights, scales, and margins, typically tuned manually or through expensive black-box optimization despite their differentiability. We propose AlignedHPO, an efficient framework for tuning differentiable loss hyperparameters using gradient alignment and surrogate validation objectives. Our method employs Jacobian-vector products to reduce hypergradient computation time and memory, while differentiable metric surrogates guide optimization toward task-specific evaluation criteria. We address gradient magnitude ambiguity and stochastic instability through normalized alignment and gradient smoothing. Experiments with generalized Focal Loss and ArcFace demonstrate effective adaptation across long-tailed classification and open-set face recognition. AlignedHPO achieves competitive or improved performance compared to implicit differentiation, with nearly 12x faster hypergradient computation. These results highlight the potential of efficient surrogate-guided loss hyperparameter optimization.
Introduction
Loss function design plays a central role in deep learning across applications such as computer vision, natural language processing, and reinforcement learning. Modern loss functions often involve multiple hyperparameters, including weighting coefficients, scales, margins, and focusing factors, which influence training dynamics and final model performance. Despite their importance, these hyperparameters are typically selected through manual experimentation or expensive black-box optimization, often requiring numerous training runs.
Beyond computational cost, hyperparameter tuning complicates the development and evaluation of new training objectives. Performance improvements may depend as much on hyperparameter selection as on the loss formulation itself, while competing methods are rarely tuned with equal care. Consequently, suboptimal hyperparameter choices can bias experimental comparisons and undermine the reliability of empirical conclusions.
Many loss hyperparameters, however, are differentiable, enabling gradient-based optimization through their influence on the training trajectory. This principle underlies hypergradient and bilevel optimization methods, which have been applied to optimization parameters, loss weights, and learned training objectives. Nevertheless, efficiently tuning general loss hyperparameters toward task-specific evaluation metrics remains challenging.
Existing approaches to automatic loss optimization typically rely on differentiation through the training trajectory or implicit differentiation. The former can introduce substantial computational and memory overhead when unrolling multiple optimization steps. The latter avoids unrolling but generally assumes an approximately stationary training objective and requires expensive inverse-Hessian–vector product approximations. These limitations make both approaches costly for online hyperparameter optimization in large neural networks.
Another challenge lies in the choice of validation objective. Evaluation metrics, such as classification accuracy and ROC AUC, are often non-differentiable, while conventional validation losses may not accurately reflect the target metric. Differentiable metric surrogates offer a way to bridge this gap, allowing loss hyperparameters to be optimized toward the desired evaluation criterion.
In this work, we propose AlignedHPO, an efficient framework for surrogate-guided optimization of differentiable loss hyperparameters during training. Our method combines local gradient alignment with forward-mode automatic differentiation, using Jacobian–vector products to efficiently estimate hypergradients without unrolling training trajectories or approximating inverse Hessians. We further address limitations of direct gradient alignment, including undesirable gradient scaling and stochastic instability, through normalized similarity objectives and historical gradient smoothing.
We evaluate AlignedHPO on generalized Focal Loss and ArcFace objectives, jointly optimizing up to hundreds of hyperparameters. Experiments on long-tailed classification demonstrate improvements over fixed loss configurations and competitive performance with implicit differentiation at substantially lower computational cost. Open-set face recognition experiments further demonstrate dataset-specific adaptation while highlighting the challenges of generalization across different validation distributions.
Our main contributions are summarized as follows:
We introduce a framework for online optimization of differentiable loss hyperparameters, including nonlinear parameterizations, guided by surrogate validation objectives that approximate task-specific evaluation metrics.
We develop an efficient hypergradient computation procedure based on Jacobian–vector products, substantially reducing runtime and memory consumption compared to conventional reverse-over-reverse and implicit differentiation.
We analyze challenges of gradient alignment, including sensitivity to gradient magnitude and stochastic instability, and introduce normalized alignment and gradient smoothing to improve optimization robustness.
We demonstrate effective loss hyperparameter adaptation on long-tailed classification and open-set face recognition, investigating both computational efficiency and generalization across evaluation distributions.
Related Work
Gradient-based hyperparameter optimization. Gradient-based HPO exploits the dependence of training on hyperparameters. Early methods differentiated through complete trajectories using reversible learning [9], followed by forward- and reverse-mode differentiation [8] and truncated backpropagation [24]. Implicit differentiation instead estimates hypergradients at approximately stationary solutions, requiring inverse-Hessian–vector product approximations [18]. Recent stochastic formulations relax stationarity assumptions [13]. However, trajectory differentiation incurs substantial computational and memory costs, while implicit methods require expensive Hessian-related computations. We focus on efficient online optimization without long unrolling or stationarity assumptions.
Adaptive weighting and auxiliary learning. Related methods adapt the contributions of training examples, tasks, and auxiliary losses. Gradient similarity prevents harmful auxiliary updates [7], while OL-AUX estimates auxiliary gradient contributions [16]. Validation-based meta-gradients enable example reweighting [22], extended to nonlinear weighting by Meta-Weight-Net [25]. Auto- learns task weights through bilevel optimization [17], while AMAL adapts instance-level loss mixtures [26]. Gradient-based loss balancing has also been explored in Grad-Norm [4], while PCGrad [30] uses gradient projections to mitigate conflicting updates. These approaches primarily optimize example or task weights rather than internal loss hyperparameters.
Automatic loss design. L2T-DLF learns training losses using a teacher network and trajectory differentiation [27]. AuxiLearn uses implicit differentiation to learn nonlinear combinations of auxiliary losses and tasks [21], while AutoBalance optimizes class- and group-dependent loss parameters for imbalanced classification [14]. These methods support nonlinear objectives but incur costly trajectory differentiation or iterative inverse-Hessian approximations. In contrast, we optimize general differentiable loss hyperparameters without unrolling or Hessian estimation, demonstrating our method on Focal Loss and ArcFace [6, 15].
Efficient gradient alignment and hypergradients. Reverse-mode differentiation computes one-step hypergradients without explicitly constructing mixed-derivative matrices [8, 22]. More closely related, GraP [11] optimizes linear weights of composite pretraining losses through gradient alignment in the shared embedding space, using normalization to address gradient-scale ambiguity. Our method extends alignment to nonlinear loss hyperparameters and employs forward-over-reverse Jacobian–vector
: auxiliary losses; : unrolled steps; : inverse-Hessian iterations. FP/BP/JVP/HVP: forward/backward, Jacobian-vector, and Hessian-vector products. Target objective denotes the criterion guiding loss-parameter updates, not the evaluation metric. *Nonlinear combinations of predefined auxiliary losses. †Class-specific parameters of a cross-entropy-based loss. Costs exclude standard training and validation-gradient computation; constant factors and loss-level operations are omitted.
products for efficient computation. We additionally introduce historical gradient smoothing to improve stability under stochastic training.
Surrogate validation objectives. Differentiable surrogates enable optimization toward non-differentiable evaluation metrics. Pairwise losses, including sigmoid approximations, have been studied for AUC maximization [28], while EXACT [12] targets classification accuracy. We use AUC surrogates to guide online adaptation of training loss hyperparameters toward task-specific evaluation criteria.
Table 1 summarizes existing approaches by application, loss parameterization, optimization strategy, and computational complexity.
| Method | Application | Nonlinear param. | Tuning | Target objective | Complexity |
| OL-AUX [16] | AUX | – | Gradient alignment | Main loss | BP |
| Auto- [17] | MTL | – | Bilevel (FD) | Val. task loss | FP/BP |
| AMAL [26] | AUX | – | One-step bilevel | Val. task loss | BP |
| Meta-Weight-Net [25] | Design/HPO | – | One-step bilevel | Cross-Ent. | BP |
| L2T-DLF [27] | Design/HPO | ✓ | Unrolled bilevel | Expected metric | HVP + mixed |
| ML3 [1] | Design/HPO | ✓ | Unrolled bilevel | Meta-task loss | BP/HVP |
| AuxiLearn [21] | AUX | ✓* | Implicit diff. | Val. main loss | HVP |
| AutoBalance [14] | Design/HPO | ✓† | Implicit diff. | Balanced loss | HVP |
| AlignedHPO | Design/HPO | ✓ | Gradient alignment | Val. surrogate | JVP |
Table 1. Comparison of differentiable loss optimization methods. We summarize their applications, support for nonlinear loss parameters, tuning strategies, target objectives, and computational complexity. Complexities indicate the number of differentiation operations rather than directly comparable runtimes.
Method
We consider the optimization of a training objective , where denotes the model parameters and is a vector of differentiable loss hyperparameters. Our goal is to adapt during training such that updates of improve a validation objective . Unlike conventional hyperparameter optimization, which evaluates multiple training trajectories, our approach exploits the local dependence of the training gradient on to obtain efficient hyperparameter updates.
Core Method
Consider a training step at the current model parameters and hyperparameters . We first approximate the training objective by its first-order Taylor expansion with respect to the hyperparameters:
where denotes a local hyperparameter perturbation.
Differentiating Eq. (1) with respect to the model parameters yields
where
Thus, the columns of describe how each loss hyperparameter modifies the training gradient. Importantly, Eq. (2) provides a linear approximation to the gradient even when the original training objective depends nonlinearly on its hyperparameters.
We now consider a single SGD update,
where is the model learning rate. Let denote the validation gradient. A first-order approximation of the validation loss after this update gives
where is independent of . Consequently, the local hypergradient is
This leads to a simple hyperparameter update,
where absorbs the model learning-rate factor. The update increases the contribution of hyperparameters whose induced gradient changes are positively aligned with the validation gradient.
A direct evaluation of Eq. (6) would require computing the mixed-derivative matrix , which can be prohibitively large. Fortunately, the method only requires its vector–Jacobian product . By the symmetry of mixed derivatives,
where denotes a directional derivative along .
Equation (8) can be computed using a Jacobian–vector product (JVP), without explicitly constructing . Since forward-mode differentiation propagates a single tangent direction through the computation, its cost does not scale linearly with the number of loss hyperparameters. In practice, this amounts to one additional forward-mode differentiation pass through the relevant computation graph, enabling efficient online hyperparameter optimization even for high-dimensional .
Gradient Similarity
The update in Eq. (7) maximizes the inner product between the training and validation gradients. Although this improves their local alignment, the inner product depends on both gradient direction and magnitude. Consequently, hyperparameter optimization may favor increasing the training gradient norm rather than improving its direction.
This distinction is important because the magnitude of the training gradient directly affects the effective model update size. Even with a fixed learning-rate schedule, hyperparameter-dependent changes in the gradient norm implicitly modify the effective learning rate, potentially interfering with the intended optimization dynamics.
Since our primary objective is to identify a training direction that improves validation performance, we also consider cosine similarity:
Unlike the dot product, cosine similarity is invariant to positive rescaling of the training gradient and therefore does not directly reward larger gradient magnitudes.
Differentiating Eq. (9) with respect to gives
In particular, at ,
The expression in brackets is the component of the validation gradient orthogonal to the current training gradient. Thus, cosine-based optimization modifies the training direction without explicitly encouraging changes in its norm. Both terms in Eq. (11) are matrix–vector products of the form and can therefore be evaluated using the JVP formulation in Eq. (8).
Gradient noise and orthogonalization. Despite its desirable scale invariance, cosine-based optimization introduces an additional challenge in stochastic training. The validation gradient and the training gradient are estimated from different data distributions and typically exhibit weaker alignment than the training gradient and its own local hyperparameter derivatives.
In particular, the correlation between and can be small and highly variable across iterations, while columns of may remain strongly correlated with . As a result, the orthogonalization term in Eq. (11) can dominate the hyperparameter update.
This may produce an undesirable unlearning effect: hyperparameters are adjusted primarily to suppress components of the current training gradient, rather than to increase alignment with the validation direction. Because stochastic gradients contain substantial batch-specific variation, such adjustments may reflect transient gradient fluctuations instead of systematic differences between training and validation objectives.
Historical gradient smoothing. To mitigate this effect, we replace the instantaneous gradient used for orthogonalization with its exponential moving average (EMA):
where controls the smoothing strength and denotes the current training gradient.
The resulting surrogate similarity direction is defined as
We then compute the hyperparameter update as
Here, the historical gradient is treated as a fixed reference direction when constructing . Thus, Eq. (14) is a surrogate hypergradient rather than the exact derivative of the instantaneous cosine. The EMA reference suppresses batch-specific fluctuations and encourages hyperparameter updates relative to the persistent training direction, instead of the potentially noisy current gradient.
Optionally, the validation gradient can also be smoothed using an EMA to reduce variance in its estimated direction. The computational advantage of the method is preserved, since the final update still requires only a single product .
We investigate both dot-product and cosine-based objectives experimentally, analyzing their effects on optimization stability and downstream performance.
Validation Loss
The proposed method relies on the validation gradient to guide training. However, evaluation metrics are often non-differentiable, making them unsuitable for direct gradient-based optimization. Moreover, a conventional validation loss does not necessarily provide a locally accurate representation of the target evaluation metric.
We therefore propose to construct differentiable surrogate validation objectives that directly approximate the metric of interest. Unlike training losses, which may prioritize convenient optimization properties, these surrogates are chosen to provide informative local gradient directions for improving the evaluation metric.
For classification and ranking tasks, we consider the area under the receiver operating characteristic curve (ROC AUC). Given validation scores for positive examples and for negative examples, empirical AUC can be expressed, ignoring ties, as
where and denote the numbers of positive and negative examples. For multiclass classification, we compute the surrogate independently for each class in a one-vs-rest setting and average the resulting losses across classes.
To obtain a differentiable objective, we replace the pairwise indicator by a sigmoid approximation:
where and controls the smoothness of the approximation.
As , the surrogate approaches the pairwise misranking rate for examples with unequal scores, corresponding to . For finite , it provides nonzero gradients around the decision boundary, allowing the method to identify hyperparameter changes that improve pairwise ranking.
The validation direction is then obtained as
By constructing the validation objective around the actual evaluation criterion, we aim to improve the local relevance of the hyperparameter updates. This distinction is particularly important when the training objective and the target metric emphasize different properties of the predictions.
Experiments
In this section, we evaluate the computational efficiency of the proposed method and its effectiveness in tuning loss hyperparameters across tasks of increasing complexity. We consider a controlled toy example, closed-set classification with shared training and validation classes, and open-set face recognition, where distribution differences introduce additional instability in gradient alignment.
Computational Efficiency
We compare the computational efficiency of the proposed Jacobian–vector-product (JVP) approach against two alternative hypergradient computation methods: reverse-over-reverse automatic differentiation and implicit differentiation based on a truncated Neumann series, as employed in AuxiLearn [21] and AutoBalance [14]. Experiments use a ResNet-50 backbone with ArcFace loss, a batch size of 32, and 1,000 class-specific margin hyperparameters on an NVIDIA RTX 4060 GPU. Implicit differentiation uses five Neumann iterations. All methods share a precomputed validation gradient, excluded from the measurements. We report median computation time and incremental peak GPU memory consumption.
As shown in Figure 1a, the proposed JVP approach requires 81.4 ms, compared to 162.3 ms for reverse-over-reverse and 974.4 ms for implicit differentiation, achieving speedups of 1.99 and 11.97, respectively. JVP and reverse-over-reverse compute the same local hypergradient, so the improvement comes solely from the differentiation strategy.

Figure 1. Computational efficiency of hypergradient estimation for an ArcFace model with a ResNet-50 backbone (batch size 32). We compare implicit differentiation using a five-step Neumann approximation, conventional reverse-over-reverse differentiation, and our forward-over-reverse implementation (AlignedHPO). Lower values are better.
Similarly, Figure 1b shows that JVP requires 1,703.1 MiB, compared to 2,061.2 MiB and 3,055.9 MiB, reducing memory consumption by 17.4% and 44.3%, respectively.
The higher cost of implicit differentiation results primarily from repeated Hessian–vector products used to approximate the inverse Hessian. Unlike the local methods, it estimates the sensitivity of an approximately stationary inner solution and thus computes a different hypergradient approximation.
Overall, forward-over-reverse differentiation substantially reduces both runtime and memory requirements, making it well suited for online loss hyperparameter optimization.
Toy Example: Aligning Training and Validation Objectives
We consider a one-dimensional binary classification problem with overlapping class-conditional distributions, and , with equal class proportions. The classifier is a linear model , where and are learnable parameters.
We optimize the weighted binary cross-entropy loss,
where controls the relative class weights.
We fix the validation weight to and initialize the training weight at . We compare standard BCE (), an oracle baseline matching the validation objective (), and AlignedHPO, which automatically tunes using gradient alignment. All methods train the classifier using SGD (); AlignedHPO additionally optimizes using Adam (). Results are averaged over five runs with independently sampled training and validation sets.
Figure 2 shows the training and validation losses and the evolution of . AlignedHPO reduces the validation loss from approximately 0.55 to 0.20, matching the oracle baseline and outperforming standard BCE, which converges to approximately 0.26. Meanwhile, the learned training weight progressively increases from 0.5 toward 0.8, closely approaching the validation weight without explicitly targeting its value. This demonstrates that gradient alignment can automatically adapt an initially mismatched training objective to recover oracle-level validation performance.
Figure 2. Toy example of automatic loss hyperparameter adaptation. (a) Training loss evaluated with each method’s own class weight. (b) Validation loss evaluated with the shared weight . (c) Evolution of the training class weight . AlignedHPO automatically adapts the training objective, achieving validation performance comparable to the oracle baseline with . Solid lines show the mean over five independent runs; shaded regions indicate one standard deviation.
Long-Tailed Classification
We evaluate AlignedHPO on CIFAR-10-LT and CIFAR-100-LT [2, 5], constructed by exponentially reducing class frequencies in the training sets. We use an imbalance ratio of 100 between the most and least frequent classes, while validation and test sets remain balanced. This creates a mismatch between the training distribution and the target evaluation metric.
Parameterized Focal Loss. We consider a generalized Focal Loss with class-dependent weights and focusing parameters :
where is the batch size and is the predicted probability of the ground-truth class. We optimize hyperparameters, with class weights normalized using softmax:
The focusing parameters are constrained to be non-negative.
Compared Methods. We compare standard cross-entropy and Focal Loss with fixed against three adaptive variants: implicit differentiation (Implicit Diff.) using a truncated Neumann approximation, and AlignedHPO with dot-product and cosine-based gradient alignment.
All adaptive methods optimize the same class-specific Focal Loss hyperparameters using the differentiable ROC AUC surrogate introduced in § 3.3. We use sigmoid temperatures of for CIFAR-10-LT and for CIFAR-100-LT. The balanced validation sets guide hyperparameter optimization, enabling a fair comparison of different hypergradient estimation strategies.
Training Details. We use a ResNet-18 [9] backbone, trained for 200 epochs using SGD with momentum 0.9 and an initial learning rate of 0.1. The learning rate is multiplied by 0.01 at epochs 160 and 180. We use a batch size of 128 for all methods except implicit differentiation, which uses 96 due to its higher memory requirements. The Focal Loss parameters are initialized with uniform class weights () and zero focusing parameters (), corresponding to standard cross-entropy.
Results and Discussion. Table 2 reports test classification accuracy on both benchmarks.
| Method | CIFAR10-LT | CIFAR100-LT |
| Cross-Entropy | 71.33 | 39.92 |
| Focal () | 72.39 | 39.83 |
| Implicit Diff. | 74.53 | 40.48 |
| AlignedHPO (Dot) | 74.79 | 40.65 |
| AlignedHPO (Cosine) | 74.41 | 42.00 |
Table 2. Classification accuracy (%) on CIFAR10-LT and CIFAR100-LT. Higher is better.
In our experiments, all adaptive methods outperform the fixed-loss baselines, demonstrating the effectiveness of surrogate-guided hyperparameter optimization. AlignedHPO improves upon standard cross-entropy by up to 3.46 and 2.08 percentage points on CIFAR-10-LT and CIFAR-100-LT, respectively. It also achieves slightly better results than implicit differentiation, while offering substantially lower computational cost (§ 4.1).
The two alignment objectives exhibit different behavior. Dot-product alignment performs slightly better on CIFAR-10-LT (74.79% vs. 74.41%), whereas cosine alignment achieves a larger improvement on CIFAR-100-LT (42.00% vs. 40.65%). This suggests that controlling gradient magnitude may be more beneficial in more challenging classification settings.
Figure 3 illustrates the evolution of learned hyperparameters on CIFAR-10-LT. We expect larger class weights for rare classes to compensate for imbalance, and larger focusing parameters for frequent classes to suppress their easier examples. The learned parameters partially follow this intuition: decreases substantially, while increases, reducing the contribution of the most frequent class. Meanwhile, increases, although the rarest class retains a weight close to its initialization. The focusing parameters for classes 5 and 9 remain close to zero.
Figure 3. Evolution of the effective Focal Loss parameters (top) and (bottom) for selected classes on CIFAR-10-LT. We compare AlignedHPO, its cosine variant, and implicit differentiation.
Notably, both AlignedHPO variants produce smooth and similar hyperparameter trajectories, whereas implicit differentiation exhibits substantially larger fluctuations and occasional sharp changes. This indicates more stable hyperparameter adaptation with the proposed local gradient alignment approach, despite its considerably lower computational cost.
Open-Set Face Recognition
We further evaluate AlignedHPO on open-set face recognition, where training and evaluation identities are disjoint. Unlike closed-set classification, this introduces an additional mismatch between training and validation objectives, making hyperparameter optimization more challenging.
We train models on CASIA-WebFace [29] and evaluate verification accuracy on LFW [10], AgeDB-30 [20], and CFP-FP [23], which capture variations in imaging conditions, age, and pose, respectively.
Parameterized ArcFace Loss. We adopt ArcFace as the training objective, which introduces an additive angular margin into normalized softmax classification:
where denotes the angle between the normalized embedding and class weight, is the angular margin, and is the logit scaling factor. We treat these parameters as differentiable hyperparameters and optimize them during training.
Validation Objective. Face verification relies on similarity scores between image pairs rather than class predictions.
We compute cosine similarity between normalized embeddings and use the differentiable binary ROC AUC surrogate from § 3.3 to guide hyperparameter optimization. This allows the classification-based training loss to be adapted toward the verification objective.
Dataset-Specific Optimization. We consider three variants of AlignedHPO, independently tuned using validation pairs from LFW, AgeDB-30, and CFP-FP. All variants share the same CASIA-WebFace training data and differ only in the validation distribution used for tuning. We evaluate both dot-product and cosine-based alignment against standard ArcFace with fixed hyperparameters (, ).
Training Details. We use a MobileFaceNet backbone [3] trained for 18 epochs with SGD, Nesterov momentum of 0.9, and weight decay of . The initial learning rate is 0.01 and is multiplied by 0.1 at epochs 6, 11, and 16.
Since the ArcFace scale parameter strongly affects the training gradient magnitude, we normalize the gradients using an exponential moving average (EMA) of their norm, with a smoothing factor of 0.9. This reduces variations in the effective learning rate caused by hyperparameter adaptation and ensures more comparable optimization dynamics across methods.
Results and Discussion. Table 3 reports verification accuracy on all three benchmarks.
| Method | LFW | AgeDB | CFP-FP |
| ArcFace (original) | 97.68 | 86.37 | 87.34 |
| AlignedHPO – Dot product | |||
| AlignedHPO (LFW) | 97.15 | 84.33 | 86.63 |
| AlignedHPO (AgeDB) | 96.98 | 85.87 | 79.44 |
| AlignedHPO (CFP-FP) | 96.68 | 82.45 | 89.26 |
| AlignedHPO – Cosine similarity | |||
| AlignedHPO (LFW) | 96.73 | 82.40 | 84.01 |
| AlignedHPO (AgeDB) | 97.13 | 85.45 | 81.97 |
| AlignedHPO (CFP-FP) | 96.68 | 81.58 | 88.17 |
Table 3. Face verification performance (%) on LFW, AgeDB-30, and CFP-FP. Each AlignedHPO variant is tuned using the corresponding validation dataset. The retrained variants use the parameters identified during hyperparameter optimization.
The results demonstrate dataset-specific adaptation of the learned loss hyperparameters. For both alignment objectives, tuning on CFP-FP achieves the highest CFP-FP accuracy among adaptive variants, while tuning on AgeDB-30 provides the best AgeDB-30 performance. A similar trend is observed on LFW with dot-product alignment. This indicates that AlignedHPO can adapt the training objective to the characteristics of the target validation distribution, even when training and validation identities are disjoint.
However, the original ArcFace baseline remains superior on LFW and AgeDB-30. The main improvement is observed on CFP-FP, where dot-product alignment achieves 89.26%, exceeding the baseline by 1.92 percentage points. These results suggest that optimizing hyperparameters for a single validation distribution can improve performance on that target but may reduce generalization to other verification benchmarks.
Another possible limitation is the mismatch between the validation objective and evaluation metric: we optimize a differentiable ROC AUC surrogate but evaluate verification accuracy. Although related, these metrics are not equivalent, potentially limiting the effectiveness of hyperparameter adaptation.
Future work could address these limitations by jointly optimizing hyperparameters across multiple validation datasets and exploring differentiable accuracy surrogates [12]. These extensions may improve the robustness and generalization of the learned training objective.
Limitations
AlignedHPO has several limitations. First, its local, single-step approximation neglects the long-term effects of hyperparameters on training, potentially limiting effectiveness when delayed effects are important.
Second, the method depends on validation gradient quality. Differences between training and validation distributions can introduce noise and instability. Although gradient smoothing mitigates these effects, our open-set experiments show that dataset-specific adaptation does not necessarily generalize across evaluation benchmarks.
Finally, performance depends on how well the differentiable validation objective approximates the target metric. In particular, our ROC AUC surrogate does not directly optimize verification accuracy. Future work could explore multi-dataset validation objectives, alternative metric surrogates, and extensions beyond local hypergradient estimation.
Conclusion
We introduced AlignedHPO, an efficient framework for online optimization of differentiable loss hyperparameters. By combining local gradient alignment with Jacobian–vector products, the proposed method avoids expensive trajectory differentiation and inverse-Hessian approximations while supporting nonlinear loss parameterizations.
We further investigated dot-product and cosine-based alignment and introduced gradient smoothing to improve optimization stability. Experiments demonstrate substantial reductions in hypergradient computation time and memory, along with competitive or improved performance on long-tailed classification and open-set face recognition. Our results also highlight the potential of differentiable metric surrogates for guiding loss hyperparameter optimization, providing a practical foundation for automated loss adaptation during deep network training.
LLM Usage Statement
OpenAI ChatGPT was used to assist with scientific writing and editing, related-work analysis, code development, and experimental plotting. All technical contributions, experimental results, and conclusions were developed and verified by the authors.
References
- [1]Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4161–4168. IEEE, 2021. 3
- [2]Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. 2019. 6arxiv.org/abs/1906.07413
- [3]Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. Mobilefacenet: Efficient cnns for accurate real-time face verification on mobile devices. In Chinese conference on biometric recognition, pages 428–438. Springer, 2018. 8arxiv.org/abs/1804.07573
- [4]Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794–803. PMLR, 2018. 2
- [5]Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 9260–9269. IEEE, 2019. 6arxiv.org/abs/1901.05555
- [6]Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019. 2DOI
- [7]Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity, 2018. 2arxiv.org/abs/1812.02224
- [8]Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International conference on machine learning, pages 1165–1173. PMLR, 2017. 2arxiv.org/abs/1703.01785
- [9]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6arxiv.org/abs/1512.03385
- [10]Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report, 2008. 7
- [11]Ivan Karpukhin and Andrey Savchenko. When losses align: Gradient-based composite loss weighting for efficient pre-training. arXiv preprint arXiv:2605.07756, 2026. 2arxiv.org/abs/2605.07756
- [12]Ivan Karpukhin, Stanislav Dereka, and Sergey Kolesnikov. Exact: How to train your accuracy. Pattern Recognition Letters, 185:23–30, 2024. 3, 8DOI
- [13]Minyoung Kim and Timothy Hospedales. A stochastic approach to bi-level optimization for hyperparameter optimization and meta learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 17913–17920, 2025. 2arxiv.org/abs/2410.10417
- [14]Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Autobalance: Optimized loss functions for imbalanced data. pages 3163–3177, 2021. 2, 3, 5arxiv.org/abs/2201.01212
- [15]Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 2
- [16]Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning, 2019. 2, 3
- [17]Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.03091, 2022. 2, 3arxiv.org/abs/2202.03091
- [18]Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics, pages 1540–1552. PMLR, 2020. 2arxiv.org/abs/1911.02590
- [20]Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pages 1997–2005. IEEE, 2017. 7DOI
- [21]Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. In International Conference on Learning Representations, 2021. 2, 3, 5arxiv.org/abs/2007.02693
- [22]Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334–4343. PMLR, 2018. 2arxiv.org/abs/1803.09050
- [23]Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV), pages 1–9. IEEE, 2016. 7DOI
- [24]Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back-propagation for bilevel optimiza- tion. In The 22nd international conference on artificial intelligence and statistics, pages 1723–1732. PMLR, 2019. 2
- [25]Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. 2019. 2, 3arxiv.org/abs/1902.07379
- [26]Durga Sivasubramanian, Ayush Maheshwari, Pradeep Shenoy, Ganesh Ramakrishnan, et al. Adaptive mixing of auxiliary losses in supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9855–9863, 2020. 2, 3
- [27]Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, and Tie-Yan Liu. Learning to teach with dynamic loss functions. 2018. 2, 3arxiv.org/abs/1810.12081
- [28]Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey. ACM computing surveys, 55 (8):1–37, 2022. 3arxiv.org/abs/2203.15046
- [29]Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7923, 2014. 7arxiv.org/abs/1411.7923
- [30]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. pages 5824–5836, 2020. 2arxiv.org/abs/2001.06782

