ExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale

Introduction

Hyperparameter optimization is universally practiced and systematically misframed. Machine learning (ML) practitioners regularly apply Bayesian optimizers, random search, and bandit-based schedulers to escape hyperparameter configuration space as efficiently as possible in search for a single optimal point and discarding the surrounding topology. Developers may opt to bypass hyperparameter tuning altogether, relying on default model settings; a practice that has been called for deprecation [2]. That topology encodes crucial scientific information [23] no optimizer is designed to preserve including the width of the optimal basin, the location of instability boundaries, and the structure of parameter interactions. A complete landscape reveals topological features (e.g., instability cliffs, lower variance secondary optima, interaction ridges, etc.) otherwise invisible to any optimizer by design.

Loss landscape analysis in weight space has demonstrated that the geometry of neural network optimization surfaces profoundly shapes generalization and training dynamics [12, 8]. The analogous program in hyperparameter space remains largely unpursued. Fitness landscape analysis [26, 17], developed in evolutionary computation, offers a mature methodological toolkit, but has never been applied systematically to ML hyperparameter spaces at scale. Large-scale hyperparameter tuning initiatives have been previously undertaken, (e.g., [24]) however they have failed to produce fine-grained resolution maps utilizable towards landscape-level scientific analysis. The computational expense of exhaustive configuration evaluation has historically precluded the systematic production of fine-grained hyperparameter landscapes [7, 13], leaving a class of scientifically rich artifacts (i.e., dense, ground-truth records of the full performance surface) effectively absent from the community’s empirical toolkit.

EXATUNE as a highly-parallelized and exhaustive form of grid search, was developed to address this gap. Our proposed system translates a declarative YAML specification into a SLURM job array, exhaustively evaluates every configuration on a discretized grid using High Performance Computing (HPC) infrastructure, and aggregates results into analysis-ready Parquet files. We present a formalized suite of landscape metrics adapted from fitness landscape analysis for application to the hyperparameter grid setting enabling the characterization of the topology of each performance surface. The result is a new artifact class: the hyperparameter landscape map, a dense, ground-truth record that can serve as a scientific instrument for the broader community.

Contributions.

1. EXATUNE: HPC infrastructure for exhaustive landscape generation. EXATUNE translates a YAML configuration into a SLURM job array, evaluates every grid point to completion with $K$-fold cross-validation, and aggregates results into reproducible Parquet benchmarks. The system enforces deterministic seeds, tolerates node failures gracefully, and is portable across HPC environments.

2. A benchmark collection of complete hyperparameter landscape maps. We release 8 exhaustive landscapes (285,080 total configurations) spanning SVMs, random forests, XGBoost, ridge regression, and $k$-nearest neighbors across six classification and two regression benchmarks, hosted on Hugging Face with Croissant machine-readable metadata for one-line loading.

3. An analytical framework for landscape characterization. We formalize five landscape metrics adapted to hyperparameter space: Moran’s $I$ for spatial autocorrelation, a ruggedness index, local optima enumeration, fitness-distance correlation, and variance-based parameter importance. Complete mathematical formalization appears in Appendix A.

4. Empirical findings on what approximate methods miss. We compare EXATUNE to HPO methods (Bayesian optimization, Optuna, Hyperband, and Random Search) under standard trial budgets, demonstrating that these methods systematically fail to detect instability regions, secondary optima, and parameter interaction effects visible in complete landscapes.

The paper is organized as follows: Section 2 situates ExaTune within the three research communities it spans and articulates what it contributes beyond existing work; Sections 3–4 detail the system architecture and the landscape metric formalization; Sections 5–6 present the benchmark landscape collection and empirical analysis, including a systematic comparison against approximate HPO methods; finally Section 7 discusses implications for practice and directions for future work.

Related Work

Hyperparameter Optimization

The dominant framing of HPO treats the performance function $f:\Lambda\to\mathbb{R}$ as a black box to be minimized within a fixed evaluation budget. Bergstra and Bengio [4] established random search as a principled baseline, showing it dominates grid search when parameter importance is heterogeneous. Bayesian optimization replaced random sampling with model-guided search via Gaussian processes [25] and tree-structured Parzen estimators [5, 10]. Bandit-based methods (e.g., Hyperband [14] and BOHB [7]) proposed adaptive elimination of unpromising configurations. Open-source frameworks including Optuna [1] and Ray Tune [15] have democratized these methods across distributed settings. Each approach optimizes over $\Lambda$, however, no one approach has sought to characterize it. To this end, EXATUNE produces a topographic map of the performance surface amenable to fitness landscape analysis, rather than a convergence trajectory through it (Figure 1).

Landscape Analysis & Hyperparameter Sensitivity

Fitness landscape analysis, originating in evolutionary biology [30], was formalized computationally by Weinberger [29] and systematized into a toolkit of ruggedness indices, autocorrelation functions, and fitness-distance correlation (FDC) by Stadler [26] and Malan and Engelbrecht [17]. In deep learning, loss landscape analysis has revealed that weight-space geometry shapes generalization: flat basins correlate with robustness [12], and distinct optima are often connected by near-loss paths [8]. The analogous program in hyperparameter space has been approached only indirectly: Probst et al. [21] measured performance gains from tuning across 40 learners, and Van Rijn and Hutter [27] estimated parameter importance via functional ANOVA on sparse HPO results. These studies characterize landscapes through samples as opposed to enumeration, and do not employ formal landscape metrics. ExaTune is situated at the unexplored intersection where fitness landscape analysis is applied to hyperparameter space, requiring complete ground-truth enumeration via HPC.

HPC Systems & Reproducibility. Distributed HPO frameworks [15, 3] treat configurations as candidates to eliminate (e.g., via early stopping) and their architectures are not amenable to characterization of structural features observed in exhaustive coverage. OpenML [28] and YAHPO [20] provide surrogate-based landscape approximations but not ground-truth enumeration. Reproducibility studies [9, 16, 6] have demonstrated that hyperparameter sensitivity dominates benchmark variance, yet the field has lacked the empirical foundation to address this at the landscape level. ExaTune is proposed to produce and encourage the generation of these scientific artifacts.

ExaTune System

Architecture

The ExaTune pipeline has five sequential phases (Figure 2). In Phase I, a YAML configuration is validated via Pydantic and expanded into an exhaustive Cartesian grid. In Phase II, one SLURM job script is generated per grid point via a Jinja2 template and submitted as a job array. In Phase III, each HPC worker independently estimates performance via $K$-fold cross-validation. In Phase IV, individual JSON results are merged into a columnar Apache Parquet store. In Phase V, the aggregated surface is analyzed with the landscape metric suite (Section 4). The system is designed for reproducibility (deterministic seeds propagated from experiment to job level), scalability (SLURM job arrays across hundreds of nodes), and portability (configurable module loading). Full implementation details appear in Appendix § 9.

Grid Enumeration and Evaluation

Given $d$ hyperparameters with value sets $\Lambda_1,\ldots,\Lambda_d$ of sizes $m_1,\ldots,m_d$, the full configuration space is the Cartesian product

Parameters may be specified on linear or logarithmic scales; each configuration receives a unique 128-bit MD5 digest of its sorted key–value pairs for collision-free lookup across distributed result files. Every grid point is a mandatory evaluation: unlike HPO frameworks, EXATUNE performs no pruning.

Each worker evaluates a single configuration $\lambda^{(i)}$ via $K$-fold cross-validation:

where $h_{\lambda,D}$ is the model fitted with hyperparameters $\lambda$ on training set $D$, $D_j$ is the $j$-th hold-out fold, and $\mathcal{M}$ is the evaluation metric (e.g., macro-F1, AUROC, RMSE). On failure, the worker writes a structured error record, preserving full-grid integrity for downstream analysis.

Hyperparameter Landscape Metrics

EXATUNE characterizes each performance surface via five metric families: smoothness (Moran’s $I$ [18], ruggedness index), multimodality (local optima density, funnel index), global navigability (fitness-distance correlation), and parameter attribution (variance-decomposition importance, pairwise interaction strength). All metrics are computed over a grid graph $\mathcal{G}=(\Lambda_G,E)$ in which two configurations are adjacent iff they differ in exactly one parameter by a single rank step; the empirical performance surface $\hat{f}:\Lambda_G\to\mathbb{R}$ is defined by Eq. (2). Complete formal definitions appear in Appendix § 8; Table 1 summarizes all metrics and their strategy implications.

Metric	Symbol	Range	Interpretation / Strategy Guidance
Moran’s $I$	$I$	$(-1, 1)$	$I \to1$: smooth; local/BO search viable. $I \approx-1/(n-1)$: random surface; exhaustive needed.
Ruggedness index	$R$	$[0, 1]$	$R \to0$: smooth; $R \to1$: near-random local variation.
Optima density	$\delta_O$	$[0, 1]$	Large $\delta_O$: many local attractors; local search unreliable.
Funnel index	$\phi_\varepsilon$	$[0, 1]$	$\phi_\varepsilon\to1$: broad optimum basin; $\phi_\varepsilon\to0$: narrow peak.
Fitness-distance corr.	FDC	$(-1, 1)$	$> 0.15$: big valley; local search works. $< -0.15$: deceptive landscape.
Eta-squared importance	$\eta_k^2$	$[0, 1]$	Fraction of performance variance explained by parameter $k$ alone.
Interaction strength	$\Omega_{k_1,k_2}$	$[0, 1]$	$\Omega> 0$: joint tuning of $(k_1,k_2)$ required; parameters non-separable.

Spatial autocorrelation (Moran’s $I$)

Spatial autocorrelation (Moran’s $I$). Let $W=[w_{ij}]$ be the symmetric adjacency matrix of $\mathcal{G}$, $S_0=\sum_{ij}w_{ij}$, $n=|\Lambda_G|$, and $\delta_i=\hat{f}(\lambda^{(i)})-\bar{f}$. Moran’s $I$ [18] measures the spatial dependence of

$\hat{f}$ with respect to grid adjacency:

$I \approx1$ indicates a smooth landscape where local search is viable; $I \approx\mathbb{E}[I]$ indicates spatial randomness; $I \ll\mathbb{E}[I]$ indicates a checkerboard landscape for which exhaustive search is essential.

Ruggedness. The ruggedness index measures local heterogeneity relative to global spread:

$R \approx0$ indicates a smooth landscape; $R \approx1$ indicates maximally rugged local variation. $R$ and $I$ are complementary: $I$ captures the direction of local dependence while $R$ measures its magnitude.

Local optima, FDC, and parameter importance. Optima density $\delta_O = |\mathcal{O}|/n$ (where $\mathcal{O}$ is the set of strict local maxima) and funnel index $\phi_\varepsilon$ quantify multimodality. Fitness-distance correlation (FDC) measures whether high-performing configurations cluster around the global optimum: FDC $> 0.15$ indicates a “big valley” structure amenable to gradient-following, while FDC $< -0.15$ indicates a “deceptive” landscape. Parameter importance $\eta_k^2$ (one-way ANOVA effect size) decomposes performance variance across individual parameters, and interaction strength $\Omega_{k_1,k_2}$ quantifies variance unexplained by main effects alone—directly motivating joint hyperparameter tuning.

Benchmark Landscape Collection

Models, Hyperparameter Grids, & Datasets

We generate complete landscape maps for five model families widely used in tabular ML: support vector machines (SVM), random forests (RF), gradient-boosted trees (XGBoost and scikit-learn GBR for regression), ridge regression, and $k$-nearest neighbors (KNN). For each model, we select the $d \in\{2,\ldots,7\}$ hyperparameters most frequently tuned in practice and define a grid with resolution chosen so that $|\Lambda_G| \le1.2 \times10^5$ per landscape to keep exhaustive evaluation tractable on a standard HPC allocation. Full grid specifications are provided in Appendix D.

The collection spans eight publicly available benchmark tasks: six classification datasets (MNIST [28], UCI Adult, Gestational Diabetes, Maternal Risk, Titanic, and Wine) and two regression datasets

(Diabetes and California Housing). Landscapes are evaluated with 5-fold cross-validation; classification landscapes use weighted F1 as $M$, and regression landscapes use negative MSE. The complete collection is hosted on HuggingFace with Croissant machine-readable metadata; each landscape is stored as a single Parquet file with one row per evaluated configuration. Table 2 reports the three primary landscape metrics for all eight landscapes. The collection totals 285,080 fully-evaluated configurations across 8 model–dataset pairs.

Model	Dataset	Task	$d$	$|\Lambda_G|$	$\mathcal{I}$	$R$	FDC	$\delta_O$	$\phi$
Classification†
SVM	MNIST	C	2	1,000	0.966	0.035	0.621	—	—
Random Forest	UCI Adult	C	6	70,200	0.848	0.120	0.398	0.024	0.216
XGBoost	Gestational Diabetes	C	7	4,862	0.718	0.272	0.113	0.105	<0.001
XGBoost	Maternal Risk	C	7	4,862	0.869	0.065	−0.120	0.293	0.086
XGBoost	Titanic	C	7	112,320	0.728	0.223	−0.013	0.052	0.127
KNN	Wine	C	5	32,040	0.692	0.458	0.229	0.353	0.020
Regression‡
Ridge Regression	Diabetes	R	5	8,316	0.626	0.006	0.019	0.007	0.571
Gradient Boosting	Housing	R	4	51,480	0.825	0.087	0.186	0.003	0.545
Total				285,080

Experiments

Landscape Characterization

We apply the full metric suite (§ 4) to each landscape in the benchmark collection; results appear in Table 2. Figure 3 shows representative heatmaps illustrating the diversity of landscape topologies. Several patterns emerge. Low-dimensional models are substantially smoother: the SVM on MNIST with $d = 2$ (C and $\gamma$) exhibits the smoothest landscape in the collection ($\mathcal{I} = 0.966$, $R = 0.035$, FDC = 0.621), consistent with a near-convex underlying optimization problem and a strong big-valley structure. Categorical and discrete hyperparameters introduce ruggedness: KNN on Wine with algorithm-selection and weight-type hyperparameters depicts the most rugged landscape ($R = 0.458$, $\mathcal{I} = 0.692$), exhibiting substantial local variation relative to global spread. XGBoost landscapes are topologically diverse: on Gestational Diabetes and Titanic, FDC is near zero or mildly negative (deceptive, Figure 1), indicating that proximity to the global optimum is not a reliable guide for local search, while on UCI Adult (Random Forest) a strong positive FDC = 0.398 confirms a well-structured big valley. Regression landscapes differ qualitatively: Ridge Regression on Diabetes has FDC $\approx0.019$ indicative of a near-flat relationship between configuration distance and performance, suggesting that regularization strength dominates performance uniformly across the grid rather than concentrating high performance near a single basin; a moderately high $\mathcal{I} = 0.626$ suggests local variability, qualitatively corroborated through visualization (Figure 3, top-middle). These findings align with and extend the tunability analysis of Probst et al. [21], which identified SVMs and regularized linear models as exhibiting very different sensitivity profiles, providing a landscape-level explanation for that observation. Funnel structure and multimodality to quantify navigability: Regression landscapes exhibit the broadest funnels ($\phi\ge0.54$) and fewest local optima ($\delta_O \le0.007$), consistent with wide high-performance basins, while KNN on Wine illustrates the $\,^{\dagger}$Summed over 4 of 7 datasets with complete energy records.

opposite extreme ($\delta_O = 0.353$, $\phi= 0.020$): maximal local-optima density paired with a narrow global peak. The starkest case is XGBoost on Gestational Diabetes, where $\phi< 0.001$ indicates the global optimum occupies essentially a single grid configuration (Figure 1) that approximate methods at standard budgets are statistically unlikely to locate.

What Approximate Methods Miss

We compare four HPO methods with EXATUNE: Random Search, Bayesian Optimization, Hyperband, Optuna (TPE) [1] at a fixed 100-trial budget across all 8 benchmark landscapes (SVM/MNIST comparison data forthcoming; see Appendix § 12). Table 3 summarises coverage, performance gap, energy consumption, and CO$_2$ emissions.

Method	Coverage	$\Delta$score	Energy (kWh)	CO$_2$ (g)
Random Search	0.09–2.1%	0.0022	0.003	0.18
Bayesian Opt.	0.09–2.1%	0.0015	0.002	0.15
Hyperband	0.09–2.1%	0.0011	<0.001	0.05
Optuna (TPE)	0.09–2.1%	0.0011	0.002	0.12
EXATUNE	100%	—	3.75$^{\dagger}$	558.8$^{\dagger}$

At a 100-trial budget, all approximate methods explore between 0.09% (Titanic, $|\Lambda_G| = 112{,}320$) and 2.06% (Gestational Diabetes and Maternal Risk, $|\Lambda_G| = 4{,}862$) of each landscape. Despite this low coverage, the best discovered configuration is close to the global optimum in absolute performance: the mean gap across 7 datasets ranges from 0.0011 (Hyperband, Optuna) to 0.0022 (Random Search) in the native score metric. However, near-optimality of the returned configuration does not imply characterization of the landscape; approximate methods by construction explicitly discard low-performing regions and thus cannot detect instability cliffs, secondary optima, or interaction structure that are only visible from complete enumeration. Importantly, the energy overhead of exhaustive evaluation is substantial: EXATUNE requires 3.75 kWh and emits 558.8 g $\mathrm{CO}_{2}$ for 4 landscapes, compared to <0.003 kWh per run on the same datasets; we contextualise this cost in § 7.

Coverage Analysis

Coverage is strictly determined by the ratio of the fixed trial budget to the total grid size $|\Lambda_G|$. At a budget of 100 trials, coverage spans a 23-fold range across the benchmark collection: from 2.06% for the two smallest landscapes (Gestational Diabetes and Maternal Risk, $|\Lambda_G| = 4{,}862$) to 0.09% for the largest (Titanic, $|\Lambda_G| = 112{,}320$). The remaining 98–100% of each landscape is unobserved by any approximate method, regardless of acquisition strategy. This unobserved fraction is not topologically neutral: regions of low gradient magnitude and broad optima basins are structures characterised by $I$, $R$, and FDC and systematically undersampled because approximate methods concentrate evaluations near high-performance regions. Exceptionally, random search samples within these regions by chance alone. Full enumeration guarantees observation of all topological features.

Scaling Analysis

Table 3 and Appendix E report wall-clock times for all landscapes. EXATUNE run times span several orders of magnitude across the collection: 23 minutes for Gestational Diabetes ($|\Lambda_G| = 4{,}862$, XGBoost, 5-fold CV) up to 55 hours 50 minutes for Titanic ($|\Lambda_G| = 112{,}320$, XGBoost, 5-fold CV). Figure 4 illustrates both the scaling behaviour and carbon emissions landscape for producing exhaustively enumerated landscapes. Wall-clock time depends on both grid size and per-configuration evaluation cost: UCI Adult (Random Forest, $|\Lambda_G| = 70{,}200$) requires 8h 26m, while Titanic (XGBoost, $|\Lambda_G| = 112{,}320$) requires 55h 50m, reflecting the higher per-node cost of Random Forest 5-fold CV on a larger feature matrix. These results demonstrate that exhaustive enumeration is tractable for $|\Lambda_G| \leq10^{5}$ on a standard academic HPC allocation using the SLURM job-array parallelism implemented in EXATUNE. For $d \geq8$ at typical per-parameter resolutions ($m_k \geq5$), $|\Lambda_G|$ exceeds $10^{6}$ and adaptive grid refinement is recommended (see Appendix § 9). All benchmark runs used the DRAC Fir cluster; per-task resource allocations are reported in Table 4 (Appendix B.5).

Discussion and Conclusion

Implications for evaluation. Hyperparameter landscape maps provide evaluative information that no HPO method is designed to produce: the width of the optimal basin, the location of instability boundaries, and the degree of parameter non-separability. A configuration atop a narrow ridge is qualitatively less reliable than one in a broad basin, even if both report identical validation performance. Landscape analysis transforms this implicit geometry into an explicit, quantifiable artifact, enabling rigorous auditing of HPO-based benchmark comparisons [6, 16].

Limitations. Exhaustive enumeration is computationally feasible only for low-dimensional HP spaces ($d \leq6$) at current grid resolutions; appropriate for classical ML modeling as opposed to deep learning. Landscapes are conditioned on fixed dataset–model–preprocessing combinations; topology may shift across splits or feature encodings. Continuous float-valued hyperparameters are discretized onto finite grids, introducing approximation error between evaluated points; the resulting dense performance tensors are nonetheless natural substrates for interpolation and imputation methods that could recover sub-grid optima without additional evaluations. Each landscape is a $d$-dimensional performance tensor whose full structure cannot be visualized beyond three dimensions: analysis must rely on 2-D slices or marginalised projections, which can obscure high-order interaction effects among non-displayed parameters and motivates dimensionality-reduction approaches tailored to landscape topology. These limitations motivate future work on adaptive grid refinement for higher-dimensional spaces, multi-fidelity landscape approximation, and extensions to neural architecture search.

Broader impacts. EXATUNE provides scientific value by enabling systematic landscape characterization that has previously been computationally infeasible: researchers gain access to ground-truth hyperparameter topographies that can improve benchmark validity, inform robust configuration selection, and reveal when optimizer behavior is dataset-specific rather than general. The benchmark collection is permanently reusable, meaning the carbon cost of generation is amortized across all downstream uses. The primary negative impact is the energy and emissions cost of exhaustive enumeration: for 4 of the 8 landscapes with complete energy records, EXATUNE consumed 3.75 kWh and emitted 558.8 g $CO_{2}$ (measured via CodeCarbon [22]), compared to $<0.003$ kWh and $<0.2$ g $CO_{2}$ per approximate HPO run on the same datasets—a factor of roughly 1,000–17,000× depending on landscape size. For context, the complete 8-landscape collection requires on the order of 1–2 kg $CO_{2}$, comparable to a single GPU training run for a moderately sized model. We recommend exhaustive enumeration only when a reusable ground-truth landscape is the scientific objective; standard HPO should remain the default for configuration selection in production settings.

Community contribution. EXATUNE is released as open-source software available at: https://github.com/Dick-AI-Lab/ExaTune The benchmark collection is permanently hosted on HuggingFace available at https://huggingface.co/datasets/Dick-AI-Lab/ExaTune_landscapes with a standardized schema and Crossfit metadata. We invite community contributions of additional landscapes for model families and datasets not covered in the initial release, with a documented submission protocol in the repository.

Conclusion. We have introduced ExaTune, an HPC system and benchmark collection that makes exhaustive hyperparameter landscape exploration tractable, reproducible, and scientifically accessible. By releasing complete ground-truth landscape maps and formalizing a landscape metric suite adapted to hyperparameter space, we provide the empirical foundation needed to apply fitness landscape analysis to ML at scale. The systematic evidence that approximate HPO methods miss instability regions, secondary optima, and interaction structure. These features have direct implications for deployment robustness and benchmark validity and motivates hyperparameter landscape maps as a first-class evaluation artifact in the ML toolkit.

Declaration of LLM Usage. The ExaTune software framework was developed in collaboration with Claude Code (Anthropic, claude-sonnet-4-6); assisting with the implementation, testing, and debugging of the codebase, guided by the architectural and mathematical specifications provided by the authors. All scientific decisions, experimental designs, hyperparameter grid definitions, and interpretations of results were made exclusively by the human research team.

Supplementary Material & Appendices

Full Landscape Metric Formalization

This appendix provides the complete mathematical formalization of all landscape metrics summarized in Table 1. All metrics are computed over the empirical performance surface $\hat{f} : \Lambda_{\mathcal{G}} \to\mathbb{R}$ defined by Eq. (2).

Formal Landscape Definitions

The maximum degree of any vertex is $2d$ (interior grid points); the diameter of $\mathcal{G}$ is $\sum_k (m_k - 1)$.

We write $\mathcal{N}(\lambda)$ for the open neighborhood of $\lambda$ in $\mathcal{G}$ and $\widetilde{\mathcal{N}}(\lambda) = \mathcal{N}(\lambda) \cup\{\lambda\}$ for the closed neighborhood.

Spatial Autocorrelation: Lag-$\ell$ Correlogram

Beyond the global $\mathcal{I}$ (Eq. (3)), ExATUNE computes the lag-$\ell$ autocorrelation $\mathcal{I}_{\ell}$ by replacing $W$ with the matrix $W^{(\ell)}$ obtained via BFS to depth $\ell$, yielding an empirical correlogram over distances $\ell= 1, 2, \ldots, L$. A rapid decay of $\mathcal{I}_{\ell}$ signals a short correlation length, implying that high-performing configurations are clustered in small regions.

Ruggedness: Gradient Statistics

The edge gradient across edge $e = (\lambda^{(i)}, \lambda^{(j)}) \in E$ is $g_e = |\hat{f}(\lambda^{(i)}) - \hat{f}(\lambda^{(j)})|$. The per-parameter mean gradient is

where $E_k$ are edges varying only in dimension $k$. The roughness coefficient is

measuring gradient variability relative to the performance range. Large $\varrho$ indicates a landscape with both flat plateaus and sharp instability cliffs.

Remark A.1. $\mathcal{R}$ (Eq. (4)) and $\mathcal{I}$ (Eq. (3)) are complementary: $\mathcal{I}$ captures the direction of local dependence (positive vs. negative), whereas $\mathcal{R}$ measures its magnitude. A large $\mathcal{I}$ implies a small $\mathcal{R}$, but the converse is not guaranteed when the landscape exhibits long-range trends.

Local Optima and Multimodality

Let $\mathcal{O} \subseteq\Lambda_{\mathcal{G}}$ be the set of all local optima and $f^* = \max_{\lambda} \hat{f}(\lambda)$. The optima density is $\delta_{\mathcal{O}} = |\mathcal{O}|/n$. The funnel index quantifies the concentration of high-performing configurations around the global optimum:

where $\varepsilon= 0.05$ by default. Small $\phi_{\varepsilon}$ indicates a sharp, isolated global optimum that is hard to locate without dense coverage.

Fitness-Distance Correlation

Parameter Importance via Variance Decomposition

For parameter $k$, partition $\Lambda_G$ into equivalence classes $C_k(v) = \{\lambda\in\Lambda_G : \pi_k(\lambda) = v\}$ and let $\bar{f}_k(v) = \lvert C_k(v)\rvert^{-1} \sum_{\lambda\in C_k(v)} \hat{f}(\lambda)$. The total and between-group sums of squares are

The eta-squared importance $\eta_k^2 = \operatorname{SS}_{\mathrm{B}}^{(k)} / \operatorname{SST} \in[0,1]$ is the fraction of total variance attributable to parameter $k$ alone (the one-way ANOVA effect size). Note $\sum_k \eta_k^2 \le1$, with equality iff parameters are additively independent.

Two complementary importance measures are also reported: (i) Spearman correlation $\lvert\rho_s(\pi_k,\hat{f})\rvert$ capturing monotonic associations, and (ii) normalized range $\rho_k^{\mathrm{range}} = (\max_v \bar{f}_k(v) - \min_v \bar{f}_k(v))/(\hat{f}_{\max} - \hat{f}_{\min})$ measuring the performance span across parameter levels. A composite rank is the mean of the three individual ranks, providing robustness to the pathologies of any single measure.

Pairwise Parameter Interactions

System Implementation Details

Declarative Configuration Schema

Every experiment is specified by a single YAML file parsed into a hierarchy of Pydantic models: HyperparameterSpec (one per parameter), ModelConfig, DatasetConfig, EvaluationConfig, SlurmConfig, and root ExtraTuneConfig. Pydantic enforces type constraints and cross-field invariants at load time. For example, logarithmically-spaced parameters require a strictly positive lower bound to prevent misconfigured jobs from consuming HPC allocations.

Logarithmic Grid Spacing

For multiplicative-scale parameters (e.g., learning rate), EXATUNE supports logarithmic grids. Given bounds $[\lambda_{\min}, \lambda_{\max}]$ and a points-per-decade parameter $s$, the grid points are

SLURM Execution and Portability

For each $\lambda^{(i)} \in\Lambda_G$, EXATUNE renders a Jinja2 SBATCH script encoding the full resource request, environment module loading, virtual environment activation, and a call to the worker entry point with the configuration serialized as a JSON string. The random seed for job $i$ is set to $s_0+i$, where $s_0$ is the global experiment seed, ensuring reproducibility while maintaining independence across jobs. Jobs are submitted as a SLURM array with an optional concurrency cap to avoid exhausting cluster quotas.

Clusters operated by the Digital Research Alliance of Canada (Compute Canada) intercept common Python packages (NumPy, SciPy, PyArrow, scikit-learn) with system-optimized wheels requiring module load before virtual environment creation. EXATUNE exposes a modules field in SlurmConfig; the Jinja2 template ensures module loading precedes environment activation without cluster-specific code paths in the worker.

Result Aggregation

Each worker writes result_{i:06d}.json containing $\lambda^{(i)}$, $\hat{f}(\lambda^{(i)})$, $\hat{\sigma}(\lambda^{(i)})$, per-fold scores, timing statistics, and a success flag. After all jobs complete, EXATUNE loads all JSON files into a Pandas DataFrame and writes a single Apache Parquet file with Snappy compression. This columnar format enables fast predicate pushdown for filtering by metric ranges and projection onto hyperparameter subsets. A human-readable CSV is also generated for inspection.

Cluster Configuration and Benchmark Compute Specifications

All benchmark landscapes were generated on the DRAC Fir cluster (Digital Research Alliance of Canada). Standard compute nodes feature $2 \times$ AMD EPYC 9655 (Zen 5) processors at 2.7 GHz (192 physical cores per node), 750 GB RAM, and 7.84 TB NVMe local storage; 8 high-memory nodes provide 6,000 GB RAM with $2 \times$ AMD EPYC 9654 (Zen 4) at 2.4 GHz. All benchmark jobs ran on standard compute nodes.

EXATUNE submits one SLURM array task per grid configuration (total tasks $= |\Lambda_G|$, across 285,080 tasks for the complete collection). Each task performs 5-fold cross-validation for a single $\lambda^{(i)}$ independently; tasks run concurrently up to the cluster’s scheduling limits, so elapsed wall-clock time (reported in Appendix E) is determined by the slowest job and scheduling delays rather than the sum of all task times. Table 4 reports the per-task SLURM resource allocation for each benchmark landscape.

Landscape	Model	Time limit	Memory	CPUs/task
MNIST	SVM (RBF)	12:00:00	8 GB	2
UCI Adult	Random Forest	02:00:00	8 GB	2
Pima Diabetes	Ridge Regression	06:10:00	2 GB	2
Gest. Diabetes	XGBoost	00:30:00	4 GB	2
Maternal Risk	XGBoost	00:30:00	4 GB	2
Titanic	XGBoost	12:30:00	4 GB	2
Wine	$k$-Nearest Neighbors	08:30:00	4 GB	2
Calif. Housing	Gradient Boosting	12:00:00	16 GB	4

Visualization Module

EXATUNE provides a seven-module visualization library with 13 plot types implemented in Matplotlib and Plotly. All 2-D and 3-D landscape plots annotate two reference configurations: the peak ($\operatorname{arg\,max}\hat{f}$) and the library default: the configuration corresponding to scikit-learn or XGBoost defaults, contextualizing grid-search results relative to untuned baselines. The primary plot types are: (i) heatmaps, 2-D grids with cells colored by aggregated $\hat{f}$ over marginal slices; (ii) 3-D surfaces, interpolated surfaces over two-parameter subgrids; (iii) 1-D slices, showing the marginal effect of each parameter with all others fixed at their optimal values; (iv) parallel coordinates, showing all configurations as polylines colored by $\hat{f}$ rank; (v) importance bar charts; and (vi) multi-panel dashboards integrating all of the above (Figure 5).

Benchmark Collection Details

Hyperparameter Grid Specifications

Table 5 specifies the hyperparameter grid for each model family used in the benchmark collection. All grids are fully enumerated by EXATUNE. Fixed parameters (held constant across all grid points) are shown in italics; they contribute a factor of 1 to $|\Lambda_G|$. The resulting grid size $|\Lambda_G|$ for each landscape is reproduced from Table 2.

Table 5: Hyperparameter grid specifications per model family. Type: I=integer, F=float, C=categorical. Spacing: L=linear (uniform step), Log=approximately logarithmic, —=unordered categorical or fixed. Fixed parameters (italics) are held constant; $m_k = 1$.

Model (Dataset)	Hyperparameter	Type	Range / Values	$m_k$	Spacing
SVM (MNIST)	$C$	F	$[10^{-5}, 200]$	40	Log
	$\gamma$	F	$[10^{-5}, 200]$	25	Log
	kernel	C	rbf	1	—
			Grid product:	1,000
Random Forest (Adult)	n_estimators	I	$[25, 300]$, step 25	12	L
	max_depth	I	$[3, 15]$	13	L
	min_samples_split	I	$[2, 16]$	15	L
	min_samples_leaf	I	$[1, 15]$	15	L
	max_features	C	{sqrt, log2}	2	—
	criterion	C	gini	1	—
			Grid product:	70,200
XGBoost (Gest. Diabetes, Maternal Risk)	n_estimators	I	$[25, 300]$, step 25	12	L
	max_depth	I	$[3, 15]$	13	L
	learning_rate	F	$[0.01, 0.30]$, step 0.01	30	L
	subsample	F	1.0	1	—
	colsample_bytree	F	1.0	1	—
	min_child_weight	I	1	1	—
	gamma	F	0	1	—
			Grid product (observed):	4,862
XGBoost (Titanic)	n_estimators	I	$[25, 300]$, step 25	12	L
	max_depth	I	$[3, 15]$	13	L
	learning_rate	F	$[0.01, 0.30]$, step 0.01	30	L
	subsample	F	{0.7, 0.9}	2	—
	colsample_bytree	F	{0.7, 0.9}	2	—
	min_child_weight	I	{1, 3, 5, 7, 9, 11}	6	L
	gamma	F	0	1	—
			Grid product:	112,320
Ridge Regression (Diabetes)	alpha	F	$[0.15, 9.90]$, step 0.15	66	L
	tol	F	${10^{-4}, 10^{-3}, 10^{-2}}$	3	Log
	max_iter	I	${100, 300, 500}$	3	L
	fit_intercept	C	${True, False}$	2	—
	solver	C	${auto, svd, cholesky, lsqr, sparse_cg, sag, saga}$	7	—
			Grid product:	8,316
Gradient Boosting Regressor (Housing)	n_estimators	I	$[25, 300]$, step 25	12	L
	max_depth	I	$[3, 15]$	13	L
	learning_rate	F	$[0.01, 0.30]$, step 0.01	30	L
	subsample	F	$[0.50, 1.00]$, step 0.05	11	L
			Grid product:	51,480
$k$-Nearest Neighbors (Wine)	n_neighbors	I	$[1, 89]$	89	L
	weights	C	${uniform, distance}$	2	—
	metric	C	${euclidean, manhattan, minkowski}$	3	—
	p	I	${1, 2, 3}$	3	—
	leaf_size	I	${10, 20, 30, 40, 50}$	5	L
	algorithm	C	${auto, ball_tree, kd_tree, brute}$	4	—
			Grid product:	32,040

Dataset Descriptions

Table 6 describes the eight datasets used in the benchmark collection. All datasets were preprocessed (missing value imputation, categorical encoding) prior to grid enumeration; preprocessing scripts are included in the EXATUNE repository.

Table 6: Benchmark dataset descriptions. $n$: samples as used in ExaTune (after preprocessing). $p$: feature dimensionality after encoding. F1 (W) = F1 weighted; Neg. MSE = negative mean squared error.

Dataset	Source	$n$	$p$	Task	Metric	License
MNIST	LeCun et al. [11]	70,000	784	10-class classif.	F1 (W)	CC BY-SA 3.0
UCI Adult	Kohavi (1996), UCI ML Repo.	32,561	361	Binary classif.	F1 (W)	CC BY 4.0
Pima Diabetes	NIDDK / Kaggle	2,000	8	Regression2	Neg. MSE	CC0
Gest. Diabetes	Kaggle GDM Dataset	3,525	15	Binary classif.	F1 (W)	CC BY-NC-SA 4.0
Maternal Risk	Kaggle (csafrit2)	1,014	6	3-class classif.	F1 (W)	CC BY 4.0
Titanic	Kaggle competition	1,309	273	Binary classif.	F1 (W)	CC0
Wine	UCI ML Repo. / sklearn	178	13	3-class classif.	F1 (W)	CC BY 4.0
Calif. Housing	Pace and Barry [19] / sklearn	20,640	8	Regression	Neg. MSE	Public domain

Croissant Metadata and Data Access

All landscapes are available at https://huggingface.co/datasets/Dick-AI-Lab/ExaTune_landscapes with Croissant JSON-LD metadata.

Per-Dataset HPO Method Comparison

Table 7 reports wall-clock time, grid coverage, best discovered score, energy consumption, and $CO_2$ emissions for all five HPO methods across all 8 benchmark landscapes (all approximate methods run at a fixed 100-trial budget). Energy and $CO_2$ for UCI Adult, Housing, Maternal Risk, and SVM/MNIST (EXATUNE) are not yet available (N/A); complete records are available for Diabetes, Gestational Diabetes, Titanic, and Wine. SVM/MNIST approximate method results are forthcoming and shown as placeholders.

Dataset	Method	Wall-clock	Cov.	Score	Energy (kWh)	$CO_2$ (g)
UCI Adult (RF)	Random	56m 51s	0.14%	0.8585	0.000094	0.61
UCI Adult (RF)	Bayesian	1h 9m 55s	0.14%	0.8587	0.010473	0.68
UCI Adult (RF)	Hyperband	1h 19m 12s	0.14%	0.8585	0.008144	0.53
UCI Adult (RF)	Optuna	37m 48s	0.14%	0.8585	0.006010	0.39
UCI Adult (RF)	EXATUNE	8h 26m 34s	100%	0.8592	N/A	N/A
Diabetes (Ridge)	Random	8m 45s	1.20%	0.2787	0.000062	0.004
Diabetes (Ridge)	Bayesian	14m 57s	1.20%	0.2787	0.000110	0.007
Diabetes (Ridge)	Hyperband	13m 25s	1.20%	0.2787	0.000009	0.001
Diabetes (Ridge)	Optuna	8m 38s	1.20%	0.2787	0.000042	0.003
Diabetes (Ridge)	EXATUNE	3h 9m 6s	100%	0.2787	0.003126	0.40
Gest. Diabetes (XGB)	Random	17m 11s	2.06%	0.9725	0.001790	0.116
Gest. Diabetes (XGB)	Bayesian	18m 17s	2.06%	0.9725	0.001165	0.075
Gest. Diabetes (XGB)	Hyperband	15m 2s	2.06%	0.9722	0.000502	0.032
Gest. Diabetes (XGB)	Optuna	13m 22s	2.06%	0.9722	0.001379	0.089
Gest. Diabetes (XGB)	EXATUNE	23m 40s	100%	0.9742	0.374	48.1
Maternal Risk (XGB)	Random	21m 58s	2.06%	0.8639	0.003257	0.211
Maternal Risk (XGB)	Bayesian	1h 25m 56s	2.06%	0.8639	0.012756	0.826
Maternal Risk (XGB)	Hyperband	26m 5s	2.06%	0.8659	0.001994	0.129
Maternal Risk (XGB)	Optuna	23m 43s	2.06%	0.8659	0.004697	0.304
Maternal Risk (XGB)	EXATUNE	27m 12s	100%	0.8689	0.209	0.027
Housing (GBR)	Random	3h 6m 51s	0.19%	0.8452	0.035250	2.28
Housing (GBR)	Bayesian	2h 38m 24s	0.19%	0.8458	0.027822	1.80
Housing (GBR)	Hyperband	1h 47m 55s	0.19%	0.8474	0.012342	0.80
Housing (GBR)	Optuna	40m 24s	0.19%	0.8474	0.040386	2.61
Housing (GBR)	EXATUNE	14h 26m 54s	100%	0.8477	36.61	4.703
Titanic (XGB)	Random	11m 1s	0.09%	0.8778	0.000771	0.050
Titanic (XGB)	Bayesian	27m 42s	0.09%	0.8816	0.000796	0.052
Titanic (XGB)	Hyperband	11m 13s	0.09%	0.8816	0.000170	0.011
Titanic (XGB)	Optuna	8m 29s	0.09%	0.8816	0.000316	0.020
Titanic (XGB)	EXATUNE	55h 50m 1s	100%	0.8831	1.506	193.5
Wine (KNN)	Random	8m 46s	0.31%	0.7810	0.000132	0.009
Wine (KNN)	Bayesian	29m 34s	0.31%	0.7813	0.000197	0.013
Wine (KNN)	Hyperband	10m 29s	0.31%	0.7810	0.000036	0.002
Wine (KNN)	Optuna	7m 51s	0.31%	0.7813	0.000063	0.004
Wine (KNN)	EXATUNE	12h 35m 25s	100%	0.7813	1.863	316.8
MNIST (SVM)	EXATUNE	11h 44m 31.0s	100%	0.9849	N/A	N/A