Beyond Foundation Models: PFN Priors Improve Tabular Hyperparameter Optimization

Code | Results

Introduction

Synthetic data drives the development of tabular foundation models. Since TabPFNv1 [1], every generation has come with a richer prior: TabPFNv2 added new edge types to the causal graph sampler [2], TabICLv2 added heavy-tailed random graphs and a filter on the generated datasets [3], and TabPFN-3 scaled the generated datasets to a million rows [4]. To date, these priors have mostly served one purpose: pretraining PFNs.

Conventional models, which are trained from scratch on each dataset, have no direct way to use these priors. For such models, e.g. GBDTs and DNNs, hyperparameter selection is the natural way for transferring knowledge across tasks. TabRepo demonstrates that a set of hyperparameter configurations selected on approximately 200 real datasets transfers effectively to unseen datasets [5]. Similarly, RealMLP identifies strong MLP architecture and training defaults through meta-tuning on 118 real datasets [6]. Both rely on a small, fixed collection of real datasets, which caps the scale and diversity of the meta-training tasks. This raises a natural question: can datasets sampled from a PFN prior surpass real datasets when learning transferable sets of hyperparameter configurations? To test this, we sample 4000 synthetic datasets from TabICLv2 prior, evaluate 4000 hyperparameter configurations on them, and propose a new algorithm to select $K$ best-performing configurations. On a new dataset, we only train the selected subset of $K$ configurations, so the meta-tuning gains transfer for free.

We study four efficient tabular models: LightGBM [7], LightGBM with TabPrep [8], an MLP with cosine embeddings [9], and TabPack [10]. These four models are near the efficiency-performance Pareto frontier (Figure 1). We use the TabICLv2 prior, with TabPFNv1 and TabICLv1 in ablations, and evaluate transfer on TabArena’s 51 IID datasets [11] and BeyondArena’s 142 datasets with diverse splitting protocols [12].

As a source of meta-training tasks, synthetic priors have three advantages over real datasets. First, supply is unlimited and quality is controllable. Real repositories plateau: on TabRepo’s 182 datasets, adding configurations stops helping, whereas on synthetic tasks neither the number of tasks nor the number of configurations has saturated at 4,000 (§ 5). Second, synthetic priors can be continuously improved: foundational model have shown that improving priors yields better model performance. Third, synthetic tasks rule out test-set contamination by construction. Real data needs an overlap audit: for TabArena, we removed 18 of the 200 TabRepo datasets.

The contributions of this paper are as follows:

1. Meta-tuning hyperparameters on synthetic data. We show that existing PFN priors can improve conventional tabular models through offline hyperparameter selection. Per dataset, real tasks help more, but synthetic tasks scale: given equal budget (configurations $\times$ datasets), hyperparameters selected on 4000 synthetic datasets outperform those selected on 182 real TabRepo datasets for every model and every $K$.

2. Improved tuning on real datasets. On both TabArena and BeyondArena, the selected sets of configurations outperform random sets of the same size. On TabArena, they also outperform random sets of any size, so meta-selection improves both the quality and the efficiency of downstream tuning. On BeyondArena, our LightGBM-TabPrep ranks 7th of 58 methods, ahead of TabPFN-3 and behind only recent foundation models.

3. An analysis of scaling and prior choice. We investigate how the quality of the configuration set depends on the number of synthetic tasks, the candidate pool size, and the generating prior, and derive a transfer bound and a record-count argument that motivate the observed scaling forms. Scaling number of datasets and configurations keeps paying off at 4,000, whereas the choice of prior appears secondary: even the TabPFNv1 prior is competitive with TabICLv2.

Related Work

Transfer of hyperparameter configurations. Good hyperparameter configurations typically transfer across datasets. Feurer et al. [13] warm-start Bayesian optimization with configurations that performed well on similar datasets. Later work drops the similarity and learns one fixed set for all datasets: given a table of scores for every configuration on every dataset, greedily add the configuration that most improves the average best-member score over datasets [14, 15, 16]. TabRepo scales this offline selection to 1,310 configurations on 200 datasets, and AutoGluon 1.0 ships the learned set as its default [5, 17]. RealMLP meta-learns several model components (e.g. preprocessing, activations) on 118 datasets [6]. All of this work is bounded by the supply of real data, 25 to 208 datasets. TabRepo reports a saturating effect around 150 datasets or configurations. We keep the greedy forward selection, but run it on synthetic tasks, whose supply is unlimited, and change the scoring function (§ 3.4).

A second line transfers knowledge into the optimizer rather than into a set of configurations. One line of work fits the Gaussian-process prior on evaluations from previous tasks [18, 19, 20, 21, 22]. OptFormer pre-trains a text transformer on optimization logs [23]. Recent works such as PFNs4BO and its successors, pretrain PFNs on synthetic optimization tasks [24, 25, 26, 27]. All of these search sequentially and need the surrogate model at tuning time. Our configuration set is built once offline and can be evaluated in parallel.

Synthetic priors for tabular data. Synthetic priors are the engine of tabular foundation models. Among others, the recent TabPFN, TabICL, Mitra, LimiX, Xiaomi-TabLDM, Causilo, and EXAONE Tabular are pretrained on synthetic tables alone [1, 2, 3, 4, 28, 29, 30, 31, 32, 33]. A typical synthetic prior has three parts: a directed graph over variables, an edge function that maps parents to a child, and an observation process. Graphs are random DAGs in TabPFN, Cauchy random graphs with varied connectivity in TabICLv2, and hierarchical causal models in LimiX. Edge functions started as small neural networks and grew to include decision trees and discretization in TabPFNv2, tree ensembles in TabICL, and Gaussian processes, polynomials, and products in TabICLv2 [2, 33]. Related in spirit, recent work distills a PFN into a tree ensemble or an MLP on a given dataset [34, 35]. However, distillation requires PFN during the training phase and has yet to demonstrate significant performance improvements.

Method

In this section, we present PriorTune, a method for meta-tuning strong hyperparameter configurations of conventional tabular models on synthetic data (Figure 2). Offline, we sample $N$ tasks from a synthetic prior $Q$ (§ 3.2), fit a pool $C$ of $C$ configurations once per task (§ 3.3), and greedily select $K$ of them (§ 3.4). Online, a real task

Algorithm 1 PriorTune subset selection

Require: cached validation and test predictions of every $c \in\mathcal{C}$ on synthetic tasks $D_1,\ldots,D_N$; subset size $K$; number of draws $R$; deployment rule $\mathcal{E} \in\{\mathrm{T},\mathrm{T}+\mathcal{E}\}$ equation (1)

1. Draw references $\mathcal{R}_1,\ldots,\mathcal{R}_R \sim\operatorname{Unif}(\Pi_K)$ and, for every task $i$, bootstrap resamples $D_{\mathrm{val},i}^{(1)},\ldots,D_{\mathrm{val},i}^{(R)}$ of its validation rows

2. $S_0 \leftarrow\varnothing$

3. for $k=1,\ldots,K$ do

4.     for each candidate $c \in\mathcal{C}\setminus S_{k-1}$ do

5. &nbsp;&nbsp;&nbsp;&nbsp;# deploy both with $\mathcal{E}$ on resample $r$, score on test; $h(a,b)=1_{a<b}+\frac{1}{2}1_{a=b}$

6. &nbsp;&nbsp;&nbsp;&nbsp; $\widetilde{U}(c)\leftarrow\frac{1}{NR}\sum_{i=1}^{N}\sum_{r=1}^{R}h\left(L_i^{(r)}(S_{k-1}\cup\{c\}),L_i^{(r)}(\mathcal{R}_r)\right)$

7. &nbsp;&nbsp;&nbsp;&nbsp;end for

8. &nbsp;&nbsp;&nbsp;&nbsp;$c_k\leftarrow\operatorname*{arg\,max}_{c}\widetilde{U}(c)$, ties broken by the lowest configuration id

9. &nbsp;&nbsp;&nbsp;&nbsp;$S_k\leftarrow S_{k-1}\cup\{c_k\}$

10. end for

11. return $S_K$

fits only the $K$ selected configurations and deploys them by validation, either the best one (tuned) or a greedy ensemble (tuned and ensembled).

Setup

Notation. We consider regression and classification on tabular data. We denote the synthetic data generator by $Q$ and a dataset by $D$. The loss $\ell$ is the native metric of the task, oriented so that lower is better: RMSE for regression, log-loss for multiclass, and $1-\mathrm{ROC\text{-}AUC}$ for binary classification, as in TabArena and BeyondArena. We split each dataset into train, validation, and test subsets $D_{\mathrm{train}},D_{\mathrm{val}},D_{\mathrm{test}}$.

Benchmarks. We use TabArena [11] and BeyondArena [12], which together cover 155 to 1.3 million rows, 3 to 22,215 features, and i.i.d., grouped, and temporal splits. Table 2 summarizes both suites. The main results (§ 4) use the full BeyondArena with every outer train-test split. The ablations of § 5 use TabArena-Lite, the first train-test split. Each split has up to 8 inner train-validation splits. No benchmark data enters the selection.

	TabArena	BeyondArena
Datasets	51	142
Binary / multiclass / regression	30 / 8 / 13	73 / 25 / 44
Splits: i.i.d. / grouped / temporal	51 / 0 / 0	103 / 18 / 21
Rows	748 / 6,497 / 150,000	155 / 5,000 / 1,296,316
Features	4 / 17 / 1,776	3 / 23 / 22,215

Synthetic data generation

For all main experiments, we sample tasks from the TabICLv2 prior [3]. Three settings of the prior matter most, and we keep them close to the third pretraining stage of TabICLv2. Dataset size is log-uniform between 400 and 60,000 rows. The feature count is uniform between 1 and a cap that falls from 100 to 20 as the row count grows. The sampled datasets are filtered using both of the available filters: reject if the target node of the graph shares no ancestors with feature nodes, and reject if ExtraTrees cannot beat the constant predictor. We fix the task mix at 2,000 regression, 1,400 binary, and 600 multiclass datasets with 3 to 10 classes, and split each dataset 64/16/20 into train, validation, and test, stratified for classification. Appendix § 8.§ 3 lists all settings.

Models

For our main runs, we use LightGBM [7], LightGBM with TabPrep [8], MLP with cosine embeddings [9], and TabPack [10]. We select these models for their tradeoff between training and inference speed and performance (Figure 1). Each configuration in the pool $\mathcal{C}$ with $C = 4{,}000$ is an independent draw from the model’s published search space, which we list in Appendix § 8.§ 2. TabPack is a packed ensemble of MLPs, so its unit of sampling is a single MLP configuration: it shares the pool of $C = 4{,}000$ MLP configurations, from which we select its $K = 64$ members.

Subset selection

The selection mimics the deployment rule and scores against random subsets of the same size. Following TabArena and BeyondArena, we distinguish two deployment rules: tuned (T), where the validation-best member is deployed, and tuned and ensembled (T+E), where validation builds a greedy ensemble [36]. We implement the whole procedure as a greedy forward selection algorithm: at each iteration starting from an empty set, we add a configuration, score it under the given deployment rule, and select the one that yields the highest winrate against the random subset (Algorithm 1). Appendix § 9 states the objective formally. Below we motivate each choice on a small example (see § 5.1 for ablation).

Table 1 gives a toy instance under the tuned rule: four configurations A to D, three synthetic tasks $D_1$ to $D_3$ for selection, one held-out task $D_4$ standing for a real dataset, and a subset size of $K = 2$.

Selecting on validation, scoring on test. Under both deployment rules, the model (or ensemble) is selected on the validation split and scored on the test split. Our objective mimics this, which selects configurations that are stable across validation and test. The alternative, selecting on test directly or on a single out-of-fold score as TabRepo and AutoGluon do [5], hides that instability. In Table 1, A and D are unstable: their validation and test losses disagree on most datasets, whereas B and C are stable. Both selections pick B first, then diverge: test-only selection returns $\{B, A\}$, validation-then-test returns the stable $\{B, C\}$. On the held-out $D_4$, $\{B, A\}$ deploys A and loses; $\{B, C\}$ deploys C and wins.

Comparing against the random baseline. The alternative is to use losses directly: TabRepo adds the configuration that most lowers the mean error of the best member over datasets [5]. Raw losses mix metrics and scales across tasks. In Table 1, any rule on raw losses selects

D first, since it cuts the RMSE on $D_1$ by 36 points, which outweighs every other gain in the table, but D has the worst test loss on every other dataset. Ranking objectives are preferred over regression objectives for the same reason in offline model-based optimization [37].

Stability. The third design choice is to resample both the validation split and the random reference. A single validation split decides which member is deployed: in Table 1, C beats B on the validation split of $D_1$ by one RMSE point, and another validation sample may deploy B instead. We therefore draw $R = 16$ bootstrap resamples of the validation rows, paired with $R$ independent random references $\mathcal{R}_r$; with cached predictions the cost is negligible, and averaging over references is unbiased with variance $O(1/R)$ (Proposition 4).

Configuration	$D_1$ val	$D_1$ test	$D_2$ val	$D_2$ test	$D_3$ val	$D_3$ test	$D_4$, held out val	$D_4$, held out test
A	105	103	0.28	0.36	0.60	0.45	0.38	0.47
B	96	96	0.30	0.31	0.53	0.53	0.41	0.41
C	95	97	0.34	0.34	0.55	0.55	0.40	0.40
D	60	60	0.29	0.40	0.52	0.60	0.39	0.50
Random reference	–	100	–	0.32	–	0.50	–	0.42

Main results

Setup. We evaluate on the full BeyondArena core (142 datasets) under the benchmark’s protocol: every outer train-test split, 8-fold bagging within each, native metrics, and $K = 25$ configurations per dataset (the official budget). TabPack fits its $K = 64$ members. FT and FTE denote the subsets meta-tuned for T and T+E. A random subset has 25 draws from the same search space. We rank methods by Elo, which counts pairwise wins and losses against all other methods. Figure 3 reports Elo within each BeyondArena dataset group. Appendix § 8.4.1 reports alternative metric, the median improvement over the tuned MLP.

State of the art and gains over random subsets. Under T, LightGBM-TabPrep with its FT subset has the highest Elo of all tuned methods (Figure 1). Under T+E, LightGBM-TabPrep with its FTE subset ranks 7th, above RealMLP and TabPFN-3. The 6 methods above it are pretrained foundation models: TabPFN-3.5 (fast and regular), Causilo, and 3 sizes of Kumo-Tabular [38] (Figure 10 in the appendix shows the full leaderboard). Across groups (Figure 3), LightGBM-TabPrep leads RealMLP on small and medium datasets, and RealMLP leads on tiny and large datasets and on text features. Compared to Kumo-Tabular, LightGBM-TabPrep loses on i.i.d. splits and on tiny to medium datasets, but leads it on grouped and temporal splits and on large datasets (1313 against 1183 Elo). The bars of Figure 3 show the gain of each meta-tuned subset over its random subset. LightGBM-TabPrep gains in every group, by 21 to 72 Elo under T+E and by 5 to 76 under T. The other models gain 5 to 28 Elo overall, but lose to their random subsets in some groups, most often on grouped and temporal splits. These gains cost nothing at deployment, and § 5.3 shows that, on TabArena, random subsets of any size up to the full pool of 4,000 do not close the gap.

Accuracy and inference cost. The right panel of Figure 1 plots Elo against the latency of a single-row prediction. Each PFN appears in three modes: recompute, and a key-value (KV) cache kept on the CPU or on the GPU. Up to 1313 Elo, the Pareto front consists of LightGBM, CatBoost, TabPack, and LightGBM-TabPrep. LightGBM-TabPrep with its FTE subset under T+E reaches 1313 Elo at 4.4 ms. The references with a similar Elo are 2.6 to 740 times slower: RealMLP under T+E reaches 1297 Elo at 11 ms, TabPFN-3 1279 at 243 to 3,308 ms, and TabICLv2 1215 at 102 to 1,412 ms. Above 1313 Elo, the front consists of the 3 sizes of Kumo-Tabular with a KV cache, from 1361 Elo at 81 ms to 1455 Elo at 236 ms. Meta-tuning changes only which configurations are fitted, so the latency stays within a factor of 1.8 of the random subset.

Ablations

All ablations use subsets selected under T and report the win rate of the meta-tuned subset against random subsets of the same size on TabArena (details in Appendix § 8.1.2). We use T because T+E selection runs a Caruana ensemble per candidate at every step, which is expensive. TabPack is absent: it is an online ensemble of MLP members, so the MLP configurations cover it. The main text shows $K = 200$, TabArena’s default tuning budget, and we include $K = 25, 64$ in Appendix § 8.§ 4.

Selection rule

Figure 4 ablates the choices of § 3.4. To avoid potential data leak, we evaluate out-of-fold on synthetic tasks: randomly select 3,000 datasets, deploy under T on the other 1,000, repeat 200 times. All rules choose from the same 4,000 configurations and are scored against the same random-$K$ baselines (Appendix § 8.1.2.1).

Comparing to TabRepo. The TabRepo algorithm scores the best member of a subset directly, with no validation-then-test step, so we give it one score per task on the joined validation and test rows. We replace the raw loss with TabRepo’s normalized error, $(\ell-\ell_{\mathrm{best}})/(\ell_{\mathrm{median}}-\ell_{\mathrm{best}})$ clipped to $[0, 1]$ per task [5], which removes the metric scale. From the empty set, each step adds the configuration that minimizes the mean over tasks of the best normalized error in the subset. It trails $R = 16$ for every model and $K$, by 0.010 to 0.038: even with the scale gone, the loss shape steers the selection into suboptimal regions. A naive improvement is to count instead the tasks on which a candidate improves the best member (i.e., a win rate against the current subset). The improved version beats the base in all nine cases, by 0.004 to 0.017. However, it still trails $R = 16$ by 0.014 to 0.024 for LightGBM and LightGBM-TabPrep, where scoring the best member hides configurations whose validation rank does not hold on test, and ties it on the MLP ($-0.003$ to $+0.003$).

Resampling validation and references. The plain hill-climb, with one validation split and one reference, trails $R = 16$ by 0.014 to 0.031. If we resample (16 times) just the validation, we close 60 to 84% of the gap. If we resample just the random references, we close 41 to 62%. Both matter, but validation noise appears to matter more. The win rate rises monotonically in $R$ for every model and $K$: by 0.015 to 0.030 from $R = 1$ to $R = 16$, most at $K = 200$.

Synthetic versus real meta-training data

TabRepo [5] is the standard real-data source for meta-tuning: 200 datasets, of which we drop 18 that overlap with or share a source with TabArena (listed in Appendix § 8.§ 4). We fit 16,000 configurations (our pool of 4,000 and 12,000 further draws) on the 182 datasets and select on them with the same protocol. Figure 5 compares the two sources at equal budget $B = C \cdot N$. Per task, real data is worth more at smaller budgets: up to $2^{18}$ subsets selected on the synthetic data show lower win rates, but from $2^{19}$ to $2^{20}$ on, synthetic data overtakes TabRepo. TabRepo saturates: over its last two doublings, to 16,000 configurations, its curve gains at most 0.008. We attribute this to the TabRepo size: with 182 datasets, the subsets start to overfit the source datasets. Synthetic data allows us to sidestep this, because we can continue scaling the data size indefinitely: at $2^{24}$ it ends 0.02 to 0.08 above TabRepo’s best for every model and every $K$. At $K = 25$ and 64 the crossover lies between $2^{20}$ and $2^{22}$ (Appendix § 8.§ 4).

Meta-tuning versus more random search

A cheaper alternative to meta-tuning is to fit more random configurations. Figure 6 scores random subsets of size $K'$ against the random-$K$ baseline (details in Appendix § 8.1.2.3). The mean curve rises, peaks at 0.50 to 0.53 for $K'$ between 500 and 750, and falls to 0.49 to 0.51 at $K' = 4,000: larger random subsets overfit the validation split. At $K = 25$ and 64, the mean curve peaks higher, at 0.51 to 0.60, but the meta-tuned subset still sits at least 0.07 above it (Appendix § 8.§ 4). For a practitioner with a fixed budget, the 200 meta-tuned configurations replace a random search over 4,000 configurations that fits twenty times more models and still scores lower on average.

Choice of prior

We repeat the selection on synthetic tasks from the TabICLv1 [28] and TabPFNv1 [1] priors (Appendix § 8.§ 3), with 2,000 configurations on 2,000 datasets for LightGBM, LightGBM-TabPrep, and the MLP (Figure 7). With all 2,000 configurations and tasks, the prior matters most for plain LightGBM: TabICLv2 leads at every $K$, by up to 0.09 over TabPFNv1 (0.644 against 0.551 at $K = 25$). With TabPrep, the three priors are within 0.02 of each other at every $K$. For the MLP, TabICLv2 wins at $K = 25$ and 64 and TabPFNv1 at $K = 200$. No prior wins everywhere, and the prior of TabPFNv1 appears to be competitive. We use TabICLv2, the newest prior, for the main experiments.

Scaling

Figure 8 shows the win rate over the full $(C, N)$ grid. It rises at almost every step along both axes, and neither axis has saturated at 4,000. Appendix § 9 motivates two forms: the estimation error over $N$ synthetic tasks decays as $1/\sqrt{N}$ (Proposition 2), and the number of improvements available in a pool grows as $\log C$ (Proposition 1). We fit both forms to the curves at $C = 4{,}000$ and $N = 4{,}000$ and obtain $R^2$ between 0.92 and 1.00 for $a - b/\sqrt{N}$ and between 0.90 and 0.98 for $a + b\log_2 C$, over all three models and all $K$. In Figure 17 and Figure 18 in the appendix, the curve in $C$ is steeper than the curve in $N$ for every model: at $K = 200$, a doubling of $C$ is worth 1.2 to 1.7 doublings of $N$, and the curve in $N$ tends to flatten above about 1,000 tasks. Once there are a few thousand synthetic tasks, more configurations are the better use of a fixed budget. Tasks reduce the estimation error of the objective, whereas configurations add candidates worth finding.

Selected hyperparameters

Figure 9 compares the selected configurations with the distribution that we sample them from. For LightGBM, the selected configurations prefer more expressive trees, with a small minimum data in a leaf and weak L1 regularization, together with a small learning rate: 21% of the selected learning rates are below the 0.1 quantile of the sampling distribution. For the MLP, the pattern is similar. The selected configurations prefer small learning rates, with 45% of the Muon learning rates below the 0.1 quantile, and larger embeddings: the largest dimension, 32, has a share of 21% against 14% in the sampling. For these hyperparameters, the range or the shape of the search space is thus possibly a poor fit, and a wider or reshaped space can contain better configurations.

Conclusion

We have shown that the synthetic priors built for PFN pretraining are a scalable source of meta-training tasks for conventional tabular models. PriorTune selects a subset of hyperparameter configurations on synthetic tasks alone, by win rate against random subsets under the deployment rule. We show that when scaled the synthetic tasks beat real datasets of TabRepo at equal budgets. The selected subsets beat random subsets of any size, improve LightGBM, LightGBM-TabPrep, the MLP, and TabPack on the TabArena and BeyondArena benchmarks, and make LightGBM-TabPrep the highest-ranked model in Elo on BeyondArena that is not a pretrained foundation model, above TabPFN-3 and RealMLP, at lower latency than both. The released subsets are drop-in search spaces: a practitioner replaces random search over the published space with the $K$ selected configurations.

Limitations

We do not apply PriorTune to tabular foundation models. They are pretrained on tasks from the same kind of prior, so the selection can prefer configurations that change them the least. Newer priors do not always help PriorTune: the newest prior helps plain LightGBM most but gives no clear gain for LightGBM-TabPrep and the MLP (§ 5.4). The grid is fitted once per model and search space, so a new search space needs a new grid. The synthetic tasks have at most 60,000 rows and 100 features, so the selection does not see tasks as large as the largest real datasets. The ablations use the tuned rule only, because selection for ensembles is expensive. The error bars measure variation across repetitions, not uncertainty over new datasets. Our theory motivates the forms of the scaling curves but does not prove them; the fits are empirical.

AI use statement

We used generative AI tools (Claude Code, Codex) to implement parts of the experiment and analysis code, to draft and edit parts of the text, and to search for and format references. We did not use them to generate data, develop the theory, prove claims, propose hypotheses, or design the method and experiments. The authors checked and tested all code, cross-checked every number against the experiment logs, verified all references, and reviewed all text. We take full responsibility for the content of this work.

Reproducibility statement

§ 3 and Algorithm 1 specify the selection procedure and its fixed settings ($R = 16$, tie rule, 40 Caruana steps). Appendix § 8.§ 1 gives the evaluation protocol on both benchmarks, the repetition scheme of the ablations, the compute used, and the latency measurement. Appendix § 8.§ 2 lists the complete hyperparameter search space of every model. Appendix § 8.§ 3 describes the three synthetic priors and Table 9 the exact sampling settings and seeds. Appendix § 8.§ 4 lists the TabRepo datasets excluded for overlap with TabArena. Appendix § 9 states all assumptions and gives complete proofs of the propositions. The benchmarks (TabArena, BeyondArena, TabRepo) are public. The supplementary material contains the code, the selected subsets for every model and $K$, the seeds, and the score caches of all fits, from which every table and figure can be regenerated.

Setting	TabICLv2	TabICLv1	TabPFNv1
Generator	graph_scm	mix_scm	prior_bag
Datasets	4,000	2,000	2,000
Task mix (reg/bin/multi)	2,000 / 1,400 / 600	0.4 / 0.3 / 0.3	0.4 / 0.3 / 0.3
Multiclass classes	$U\{3,\ldots,10\}$	$U\{3,\ldots,10\}$	$U\{3,\ldots,10\}$
Rows	$\operatorname{LogU}[400,60{,}000]$	$\operatorname{LogU}[1{,}024,32{,}768]$	$\operatorname{LogU}[1{,}024,32{,}768]$
Features	$U[1,\operatorname{cap}(n)]$, cap 100 → 20	$\operatorname{LogU}[4,64]$	$\operatorname{LogU}[4,64]$
Graph nodes	$\operatorname{LogInt}(2,32)$	MLP, mean layers $[1,6]$	MLP, gamma layers
Categorical share	$\operatorname{clip}(U(-0.5,1.2),0,1)$	$p=0.2$ per dataset	$p=0.2$ per dataset
Node noise	none	$\operatorname{LogNum}(10^{-4},0.3)$	$\operatorname{LogNum}(10^{-4},0.3)$
Graph filter	ancestor test	none	none
Dataset filter	ExtraTrees, $p<0.05$	ExtraTrees, $p<0.05$	ExtraTrees, $p<0.05$
Split	64 / 16 / 20	64 / 16 / 20	64 / 16 / 20
Attempts per dataset	$\le100$	$\le64$	$\le64$
Base seed	20260804	0	0

Computation

Setup

Main results

Datasets. Table 2 shows distribution of features and sample sizes of both benchmarks. For full list of datasets, their selection criteria, and description, see the original work [11, 12].

Evaluation. On BeyondArena, we use every outer split of the core benchmark. Within each outer split, every method is bagged over the 8 inner folds, and each configuration is trained once per fold. T deploys the validation-best configuration; T+E fits a 40-step Caruana ensemble on the same validation predictions. Both score the held-out test split with the native metric.

Subsets. FT subsets are selected with the T objective and FTE subsets with the T+E objective (40 Caruana steps inside the selection), both with $R = 16$ on all 4,000 configurations and 4,000 synthetic tasks. No benchmark data enters the selection. TabPack’s FTE subset is the $K = 64$ subset selected with the T+E objective on the MLP pool and deployed as one pack of those members.

Ablations

All ablations select under the T deployment rule, deploy under T, and report the win rate against a random-$K$ baseline.

Selection rule (Section 5.1)

Protocol. We run 200 repetitions. Each splits the 4,000 synthetic tasks at random into 3,000 selection tasks and 1,000 held-out tasks. Every rule selects $K$ out of 4,000 configurations on the selection tasks and is deployed under T on the held-out tasks. There it is compared with 200 random-$K$ baselines, and its win rate is averaged over them. All rules in a repetition use the same tasks and baselines. Error bars in Figure 4 are $\pm1.96$ standard errors of the win rate over the 200 repetitions.

Model	$K$	16	8	4	2	1	Val.	Ref.	Neither	Norm.	Count
LightGBM	25	0.636	0.634	0.632	0.627	0.620	0.630	0.630	0.621	0.617	0.621
LightGBM	64	0.621	0.618	0.613	0.607	0.599	0.614	0.609	0.600	0.588	0.602
LightGBM	200	0.599	0.596	0.592	0.585	0.576	0.593	0.586	0.574	0.561	0.575
LightGBM-TabPrep	25	0.636	0.636	0.633	0.629	0.622	0.632	0.631	0.623	0.618	0.622
LightGBM-TabPrep	64	0.621	0.619	0.615	0.610	0.602	0.616	0.609	0.601	0.597	0.607
LightGBM-TabPrep	200	0.603	0.599	0.594	0.588	0.579	0.597	0.588	0.577	0.571	0.588
MLP	25	0.697	0.696	0.692	0.687	0.679	0.691	0.689	0.681	0.687	0.693
MLP	64	0.669	0.665	0.661	0.654	0.646	0.664	0.655	0.645	0.658	0.672
MLP	200	0.631	0.626	0.619	0.612	0.601	0.626	0.613	0.600	0.616	0.633

Our rule. It draws, once per repetition, 16 bootstrap resamples of each task’s validation rows and 16 random-$K$ references $\mathcal{R}_r$. $R < 16$ pairs the first $R$ resamples with the first $R$ references. Validation only pairs the 16 resamples with $\mathcal{R}_1$, and reference only pairs the original validation split with the 16 references.

TabRepo algorithms. They use no validation resamples and no references. They score the best member of a subset on each dataset’s joined validation and test rows, with the native metric recomputed on the union. The first version adds, at each step, the configuration that minimizes the mean over datasets of the best normalized error, $(\ell-\ell_{\mathrm{best}})/(\ell_{\mathrm{median}-\ell_{\mathrm{best}}})$ clipped to $[0,1]$, with $\ell_{\mathrm{best}}$ and $\ell_{\mathrm{median}}$ taken over the 4,000 configurations. The second starts from the configuration with the lowest mean rank, then adds the one that improves the best member on the most tasks. Both break ties by a random configuration order drawn per repetition.

Synthetic versus real meta-training data (Section 5.2)

Cells. A cell $(C,N)$ selects $K$ of $C$ configurations on $N$ meta-training tasks by Algorithm 1 with $R=16$. Synthetic cells use our pool of 4,000 configurations and 4,000 synthetic tasks. TabRepo cells use the 182 TabRepo datasets and a bank of 16,000 configurations: our pool of 4,000 plus 12,000 further draws from the same search spaces, fitted on TabRepo and TabArena. Validation resamples are drawn once per task and shared across repetitions. Ties in validation selection and in the greedy step go to the lowest configuration id.

Repetitions. We run 2,000 repetitions, shared by all cells. A repetition permutes the configurations and the tasks once; each cell takes the first $C$ configurations and $N$ tasks of that permutation.

Evaluation. The selected subset is deployed on TabArena under $T$ and compared with the first $K$ configurations of the repetition’s permutation of our 4,000-configuration pool. Both synthetic and TabRepo cells share this baseline. Also both synthetic and TabRepo $C=K$ cells therefore have 0.5 by construction.

Reported values. Each win rate is the mean over the 2,000 repetitions, its standard error is below 0.003 in every cell and is not drawn. It measures variation across repetitions on the fixed benchmark, not uncertainty over new datasets. In Figure 5, each cell $(C,N)$ is a dot at its budget $B=C\cdot N$, so the dots above one $B$ are all cells with that budget. The line joins the best cells within each $B$.

Meta-tuning versus more random search (Section 5.3)

Meta-tuned lines. The synthetic line is the cell $C=N=4{,}000$ of Appendix A.1.2.2. The TabRepo line selects on all 182 TabRepo datasets from our pool of 4,000 configurations, with the same protocol, repetitions, and baselines. Each line is the mean win rate over the 2,000 repetitions.

Random subsets. For every $K'$, we draw 200 subsets of $K'$ configurations uniformly from the 16,000-configuration bank, independently across $K'$, and deploy each under $T$ on TabArena. Each subset is compared with all 2,000 random-$K$ baselines of the meta-tuned lines, and its win rate is averaged over them. The band therefore reflects only the draw of the random subset: we report the mean over the 200 subsets and the 95% CI band.

Choice of prior (Section 5.4)

Cells. For LightGBM, LightGBM-TabPrep, and the MLP, a pool of 2,000 configurations is fitted on 2,000 tasks (datasets) from each of TabICLv2, TabICLv1, and TabPFNv1 (Appendix A.3). Within a model, the configuration pool and the TabArena evaluation are shared, so only the priors affect the selection of the $K$ configurations. A cell $(C,N)$ selects $K$ of $C$ configurations on $N$ tasks by Algorithm 1 with $R=16$.

Repetitions and evaluation. As in Appendix A.1.2.2: 2,000 repetitions shared by all cells, each permuting the configurations and the tasks once. The selected subset is deployed on TabArena under T and compared with the first $K$ configurations of the permutation. Win rates are means over the repetitions, with standard errors below 0.003.

Scaling (Section 5.5)

Cells. The synthetic cells of Appendix A.1.2.2, with all 4,000 configurations and 4,000 synthetic tasks available: $C$ and $N$ range from $K$ to 4,000, with the same 2,000 repetitions, baselines, and reported values.

Fits. We fit $a-b/\sqrt{N}$ to the win rate against $N$ at $C=4{,}000$ and $a+b\log_{2}C$ to the win rate against $C$ at $N=4{,}000$, by least squares on the means; the $C=K$ cell, 0.5 by construction, is left out of the second fit.

Selected hyperparameters (Section 5.6)

Subset. We take the $K=200$ subset selected under T on all 4,000 configurations and 4,000 synthetic tasks. For TabPrep settings, only the selected configurations with TabPrep enabled count.

Distributions. For each hyperparameter, the sampling distribution $F$ is given in Appendix A.2. For a continuous hyperparameter, we plot the density of the selected values divided by the sampling density, $f_{\mathrm{sel}}(x)/f(x)$. The sampling distribution is then flat at 1, and a value above 1 is selected more often than it is sampled from the prior. We estimate this ratio with a Gaussian kernel (Scott bandwidth) and plot it as the coloured line. The Kolmogorov–Smirnov (KS) distance is $\sup_x\lvert\widehat{F}_{\mathrm{sel}}(x)-F(x)\rvert$, where $\widehat{F}_{\mathrm{sel}}$ is the empirical distribution function of the selected values. It is in $[0,1]$: 0 when the selected values follow the sampling distribution exactly, and larger when they move away from it. Categorical hyperparameters are shown as shares.

Compute

We report compute in A100 hours. CPU work is converted at AWS on-demand prices:² 1 A100 hour costs as much as 61.5 vCPU-hours. Table 3 summarizes the costs.

	PriorTune	PFN
Once per model	Grid: 400 (MLP); 300 (LightGBM-TabPrep); Selection: < 0.01 (T), 10 (T+E, $K = 200$)	Pretraining: 1,180 (TabICLv2); 1,440 (TabICL); 460 (TabPFNv2); not reported (TabPFN-3)
Per dataset	Fit $K = 25$: 0.1 (0.04), LightGBM proxy	Fit and predict: 0.48 (0.007), TabICLv2

PriorTune, once per model. (i) Configuration$\times$task grid. We fit MLPs with the tensorized code of TabPack [10]: one fit takes 0.1 A100 seconds on average on the synthetic tasks, so the 16 million fits of the 4,000 $\times$ 4,000 grid take 400 A100 hours. One LightGBM-TabPrep fit takes about 5 core-seconds, so its grid takes 20,000 core-hours, about 300 A100 hours. (ii) Selection. Under T, selecting a subset from the cached predictions takes seconds. Under T+E, selecting 200 configurations takes about 10 A100 hours.

PriorTune, per dataset. A new dataset fits only the $K$ selected configurations, so the cost equals that of tuning with $K$ random ones. As a proxy, we take the benchmark’s tuned LightGBM on BeyondArena-Lite (26 configurations, 8 folds, 16 vCPUs): 8 vCPU-hours per dataset on average (median 2.4), that is, 0.1 A100 hours (median 0.04).

PFNs, once. Pretraining TabICLv2 takes 24.5 H100-days, about 1,180 A100 hours [3]. TabICL takes 20 days on three A100s, 1,440 A100 hours [28]; TabPFNv2 takes about two weeks on eight RTX 2080 Ti, 2,690 GPU-hours [2], or about 460 A100 hours. TabPFN-3 does not report its pretraining compute [4].

PFNs, per dataset. In-context learning replaces fitting with one fit-and-predict pass. On BeyondArena-Lite, TabICLv2 takes 0.48 A100 hours per dataset on average (median 26 seconds). TabPFN-3 takes 0.70 GPU-hours on average (median 53 seconds).

Inference latency

Datasets and fits. We measure latency (Figure 1) on 10 BeyondArena datasets that cover its dataset groups (Table 4). For each dataset, we fix one outer train-test split. Under T, we deploy the validation-best configuration. Under T+E, we build the 40-step greedy ensemble of TabArena on the validation predictions. Ensemble members predict sequentially, and the ensemble averages their predictions. TabPack tensorizes the ensemble. In-context models use the inner training and validation splits as context and run at default settings, which include their internal ensembling: 8 estimators for TabPFN-3 and TabICLv2, and 16, 8, and 8 for the large, medium, and small Kumo-Tabular. The Elo of Figure 1 comes from the full BeyondArena.

Dataset	Task	Split	Rows	Features
credit-g	binary	i.i.d.	666	20
mice-protein	multiclass	grouped	720	77
coffee-rating	regression	temporal	2,137	12
predict-students	multiclass	i.i.d.	2,949	36
churn	binary	i.i.d.	3,333	19
emscad	binary	i.i.d.	11,640	17
video-transcoding	regression	grouped	44,897	19
sdss17	multiclass	i.i.d.	52,035	11
sf-permit-time	regression	temporal	60,784	37
kick	binary	temporal	68,012	32

Timing. For each method and dataset, the fitted model is kept in memory. We do 5 warm-up predict calls on single test rows and then time 50 calls. We report the median over calls and then over the 10 datasets. All methods run on one host. GBDTs run on 16 CPU threads. DNNs and in-context models run on one GPU.

KV cache. We time each PFN in three modes. In recompute, every call re-encodes the training context. With a key-value (KV) cache, the encoded context is computed once at fit time and kept on either the GPU or on the CPU. Table 5 lists the options. The KV cache of TabPFN-3 is quantized to int8 by default, so its cached predictions differ from recompute by up to 0.14 in probability. For TabICLv2, they differ by up to 0.015.

Table 5 | Options of the in-context models for the three timed modes. TabPFN-3 and TabICLv2 take them in the constructor (tabpfn 9.0.0, tabicl 2.2.0). Kumo-Tabular takes them in the TabArena wrapper.

Model	Recompute	KV cache on GPU	KV cache on CPU
TabPFN-3	fit_mode="fit_preprocessors"a	fit_mode="fit_with_cache" keep_cache_on_device=True	fit_mode="fit_with_cache" keep_cache_on_device=False
TabICLv2	kv_cache=Falsea	kv_cache=True	kv_cache=Trueb
Kumo-Tabular	cache_context=False	cache_context=Truec	cache_context=Truea

^aDefault. ^bTabICL has no option to keep the cache on the CPU: we move it to the CPU after fit and copy it to the GPU in every call. ^cThe wrapper keeps the cache on the CPU: we move it to the GPU once after fit.

Inference with TabPrep. For TabPrep, we compile the feature generation in Numba and make it faster in two ways: (i) the fitted feature generators become lookup tables, and (ii) we compute only the generated features that the booster (LightGBM) splits on. The compiled predictor gives identical predictions. Without compilation, LightGBM-TabPrep under T costs 44 to 51 ms per row through AutoGluon, against 0.8 to 0.9 ms compiled, 51 to 54 times slower.

Hyperparameter search spaces

Every pool is a set of independent random draws from the search space below; no configuration is tuned on any dataset. The same draws are fitted on every synthetic and real task.

LightGBM. We use the LightGBM search space of TabArena [11], shown in Table 6. Training follows the AutoGluon LGBModel semantics: up to 10,000 boosting rounds with AutoGluon’s adaptive early stopping (offset 20, rate 0.3, maximum patience 300 rounds, scaled down above 10,000 training rows) on the validation split. The pool has $C = 4{,}000$ configurations.

Parameter	Distribution
Learning rate	$\operatorname{LogUniform}[5\mathrm{e}{-3}, 1\mathrm{e}{-1}]$
Feature fraction	$\operatorname{Uniform}[0.4, 1.0]$
Bagging fraction	$\operatorname{Uniform}[0.7, 1.0]$
Bagging frequency	$1$
# leaves	$\operatorname{LogUniformInt}[2, 200]$
Min. data in leaf	$\operatorname{LogUniformInt}[1, 64]$
Extra trees	$\{\mathrm{False}, \mathrm{True}\}$
Min. data per group	$\operatorname{LogUniformInt}[2, 100]$
Categorical L2	$\operatorname{LogUniform}[5\mathrm{e}{-3}, 2.0]$
Categorical smoothing	$\operatorname{LogUniform}[1\mathrm{e}{-3}, 100]$
Max. categories to one-hot	$\operatorname{LogUniformInt}[8, 100]$
L1 regularization	$\operatorname{Uniform}[1\mathrm{e}{-4}, 1.0]$
L2 regularization	$\operatorname{Uniform}[1\mathrm{e}{-4}, 2.0]$
Max. boosting rounds	10,000
Early stopping	AutoGluon adaptive (offset 20, rate 0.3, max. 300)

LightGBM with TabPrep. The same LightGBM space, with the TabPrep feature generators of Tschalzev et al. [8] prepended to the pipeline. Following the published protocol, TabPrep is enabled for one configuration in four; an enabled configuration also draws the eight TabPrep axes in Table 7. The pool has $C = 4{,}000$ configurations and shares its LightGBM draws with the plain LightGBM pool, so the two pools differ only in preprocessing.

Parameter	Distribution
TabPrep enabled	$\operatorname{Bernoulli}(0.25)$
OpenFE max. features	$\{2000, 1000\}$
OpenFE generation seed	$\{42, 84, 168, 336, 672\}$
Cat. combination max. features	$\{100, 500\}$
Cat. combination generation seed	$\{42, 84, 168, 336, 672\}$
Random-subset feature combination # subsets	$\{50, 1, 100\}$
Random-subset generation seed	$\{42, 84, 168, 336, 672\}$
OOF target encoding generation seed	$\{42, 84, 168, 336, 672\}$
Group-by max. features	$\{500, 100, 1000\}$

MLP with cosine embeddings. The MLP is the independent member model of TabPack [10]: a feed-forward MLP with cosine numerical embeddings [9], trained with the Muon optimizer for the hidden weights and AdamW for the rest. The pool is a frozen registry of $C = 4{,}000$ draws from the space in Table 8. The seven sampled axes follow TabPack’s cosine base model; the hidden width is fixed at 384. Members are trained individually with per-member optimizer steps, up to 256 epochs, and early stopping after 16 non-improving validations, without TabPack’s online ensemble.

Parameter	Distribution
# blocks	UniformInt[1, 4]
Block width	384
Dropout rate	{0.0, Uniform[0.0, 0.5]}
Embedding dim	UniformInt[8, 32], step=4
Embedding init. scale	LogUniform[1e-2, 1e1]
Learning rate (AdamW)	LogUniform[1e-4, 5e-3]
Learning rate (Muon)	LogUniform[1e-3, 1e-1]
Weight decay	LogUniform[1e-3, 1.0]
Batch size	dataset-dependent (TabArena rule)
Max epochs	256
Early-stopping patience	16

TabPack. TabPack [10] trains 64 MLP members as one packed tensor program and combines them with the greedy online ensemble of the TabPack paper. The members are the first 64 configurations of the MLP registry above. A meta-tuned TabPack is a pack of $K = 64 configurations selected from the same pool of 4,000 MLP configurations with the T+E objective and deployed with the same online ensemble; Figure 1 and Figure 3 use this subset.

Synthetic priors

All three priors are structural causal models: a directed acyclic graph over latent variables, a random function per edge set, and an observation process that turns node values into a table. This section lists the three parts for each prior.

Graph structure

TabICLv2. A random Cauchy DAG with LogInt(2, 32) nodes. The probability of an edge from node $i$ to node $j$ is $\sigma(A + B_i + C_j)$, where $A$, $B_i$, $C_j$ are independent standard Cauchy variables: $A$ sets the overall density, $B_i$ and $C_j$ the out- and in-degree of each node. Features and the target are assigned to random nodes. A graph is resampled until the feature nodes and the target node share an ancestor.

TabICLv1. A layered MLP graph as in TabPFNv1: $num\_layers$ layers of width $hidden\_dim$, both drawn from log-scaled truncated normals with per-dataset means in [1, 6] and [5, 130], with block-wise or Bernoulli dropout of edges. In causal mode (sampled per dataset) features and target are read from hidden activations; otherwise the root causes are the features and the last layer is the target.

TabPFNv1. The same layered MLP graph, with layer count and width from gamma meta-distributions (means up to 6 and 300), $num\_causes$ root variables, edge dropout, and the same causal or non-causal read-out. With probability $1/(1 + e^w)$, $w \sim U(2, 10)$, the dataset is instead a draw from an RBF Gaussian process on Gaussian inputs [1].

Edge functions

TabICLv2. Eight function types: MLP, tree ensemble (symmetric trees, LogInt(1, 128) trees of depth 1 to 7), discretization to random centers, Gaussian process with a random kernel, linear, quadratic, EM-style cluster assignment, and product. The first three follow TabPFNv2, and the other five are new. With several parents, the prior either concatenates them and applies one function, or applies a function per parent and aggregates by sum, product, max, or log-sum-exp. Each node output is standardized, rescaled by a random feature-importance vector, and multiplied by a random node importance.

TabICLv1. With probability 0.7 an MLP layer: a random affine map followed by an activation drawn from 19 fixed functions (Identity, Tanh, LeakyReLU, ELU, ReLU, ReLU6, SELU, SiLU, Softplus, Hardtanh, sign, sine, RBF, exponential, $\sqrt{|x|}$, $1_{|x|\le1}$, $x^2$, $|x|$) or a random Fourier-feature function. Activations may differ between layers. With probability 0.3 a tree layer: an XGBoost multi-output regressor with at most 4 trees of depth at most 4, fitted on standard-normal targets and evaluated on the parent values.

TabPFNv1. A random affine map followed by one activation per dataset, shared by all layers, drawn from Tanh, Identity, or ReLU.

Transformations and filters

TabICLv2. Each scalar feature is min-max scaled and then warped by a Kumaraswamy function, with exponents drawn from LogNum(0.2, 5). Categorical features come from discretizing sub-vectors of node values into 2 to 9 categories. Outliers are clamped at 4 standard deviations, and standardized values are clipped to $[-100, 100]$. Constant columns are removed, and the feature order and class labels are permuted. Two filters reject datasets. The ancestor filter rejects a graph whose target shares no ancestor with the features. The ExtraTrees test fits 25 trees of depth 6 with bootstrapping and keeps a dataset only if their out-of-bag error beats the mean predictor in at least 95% of 200 bootstrap resamples.

TabICLv1. Gaussian noise is added after every layer, with a standard deviation drawn per dataset from LogNum($10^{-4}$, 0.3). Root causes come from a normal, a uniform, or a mixed distribution. With probability 0.2, a dataset has categorical features. Each column then becomes categorical independently, with a uniformly drawn probability, and its values are binned by rank into $\max(2,\operatorname{round}(\Gamma(1, 10)))$ categories. Outliers are clamped at 4 standard deviations, and the features and the regression target are standardized. Class labels are permuted. The original prior applies no filter. We apply the ExtraTrees test of TabICLv2, so that all three corpora are filtered alike.

TabPFNv1. Gaussian noise is added at every node, with a mean and a standard deviation drawn per node. The standard deviation comes from LogNum($10^{-4}$, 0.3). Categorical features are generated as in TabICLv1: with probability 0.2, binned by rank into $\Gamma(1, 10)$ categories. Outliers are removed, and the features and the target are z-normalized. Features are rescaled by the fraction of columns in use. For classification, class boundaries are sampled from the data, and class labels are permuted. The original prior applies no filter. We apply the ExtraTrees test of TabICLv2.

Sampling settings

Table 9 lists the sampling settings. TabICLv2 is the main prior of § 3.2. TabICLv1 and TabPFNv1 share one shape specification: dataset $i$ has the same task type, row count, feature count, and class count in both sets, so a difference between them is a difference of generator alone.

Additional results

Main results

Full leaderboard. Figure 10 shows the Elo of all 58 methods on the full BeyondArena.

Improvement over the MLP. Figure 11 repeats Figure 3 with the improvement over the tuned MLP, $100(\ell_{\mathrm{base}}-\ell)/\ell_{\mathrm{base}}$. We average across the splits of each datasets and then take the median across datasets. Unlike Elo, the improvement keeps the size of each gain [39] and does not depend on the set of compared methods. However, it depends on the choice of the baseline and averaging conceals skewness. Elo has the opposite properties, so neither metric suits every application [39]. Under T, LightGBM-TabPrep with its FT subset has the highest median (+0.71). Under T+E, Kumo-Tabular (+6.04) and RealMLP (+1.69) lead LightGBM-TabPrep (+1.34). LightGBM-TabPrep leads Kumo-Tabular on grouped splits and on large datasets.

Selection rule

Table 10 gives the win rate of every selection rule of Figure 4.

Synthetic versus real meta-training data

Figure 12 repeats Figure 5 for $K \in\{25, 64, 200\}$. At $K = 25$, TabRepo is ahead at equal budget up to $2^{20}$ to $2^{22}$ for all three models. The synthetic tasks end above its best for all three (Figure 12).

TabRepo overlap. Of the 200 TabRepo datasets we exclude 18. Ten are content-identical to or listed as overlapping with a TabArena dataset: Amazon_employee_access, APSFailure, Bioresponse, blood-transfusion-service-center, diabetes, Diabetes130US, diamonds, hiva_agnostic, jm1, splice. Eight share the source data of a TabArena dataset under a different curation: bank-marketing, churn, credit-g, houses, qsar-biodeg, wine-quality-red, wine-quality-white, wine_quality.

Meta-tuning versus more random search

Figure 13 repeats Figure 6 for $K \in\{25, 64, 200\}$.

Choice of prior

Figure 14 and Figure 15 repeat the top and bottom rows of Figure 7 for $K \in\{25, 64, 200\}$.

Scaling

Figure 16 shows the win rate over the $(C, N)$ grid, as in Figure 8, for $K \in\{25, 64, 200\}$. Figure 17 and Figure 18 show the curves along $N$ and along $C$.

Selected hyperparameters

Figure 19–Figure 21 show all hyperparameters of each model, as in Figure 9.

Theory

Setup and main propositions

Setup

Fix a model $f$, a learning algorithm $\mathcal{A}$, a candidate pool $\mathcal{C}$ of $C$ hyperparameter configurations, and a deployment budget $1 \le K \le C$. A subset is $\mathcal{S} \in\Pi_K := \{\mathcal{S} \subseteq\mathcal{C} : |\mathcal{S}| = K\}$. The goal is to choose one subset that performs well on the target distribution of real tasks, using synthetic tasks for meta-training. Performance is win rate against a uniformly random subset of the same size. All statements condition on the candidate pool.

Tasks and deployment. Let $\mathcal{P}$ and $\mathcal{Q}$ be the target (e.g. TabArena or BeyondArena benchmarks [11, 12]) and source (e.g. sampled from the TabICLv2 prior [3]) distributions over tasks. Sample a target task $\tau\sim\mathcal{P}$, which specifies a row distribution $P_\tau$. Sample the rows of $D$ independently from $P_\tau$ and split the dataset into training, validation, and test sets of prescribed sizes. For each configuration, fit

where $\xi$ collects the randomness in the dataset split and in the training of every configuration (initialization, subsampling, and any other stochastic step of $\mathcal{A}$). Each configuration is fitted once per task, and every subset that contains it, including the random reference it is compared against, reuses that fit. A deployment rule $\mathcal{E}$ takes as an input a subset (its fitted members, train and validation subsets) and outputs a single predictor. We then score it on the test set:

We require that $\mathcal{E}$ be deterministic given its inputs and that the same $\mathcal{E}$ be applied to a subset and to any subset it is compared against.

Two deployment rules. We use the two regimes of Erickson et al. [11], Purucker et al. [12]. Tuned (T) keeps the validation-best member,

Tuned and ensembled (T+E) runs $B$ steps of greedy ensemble selection with replacement on the validation split [36]. Writing $w_c$ for the share of the $B$ slots taken by configuration $c$,

The tuned regime is the special case of tuned and ensembled that puts all weight on one member.

Population objective. Let $h(a,b) = \mathbf{1}_{a<b} + \frac{1}{2}\mathbf{1}_{a=b}$. The per-task utility of a subset is

and the population objective is

A uniformly random subset scores $\mathbb{E}_{\mathcal{S} \sim\operatorname{Unif}(\Pi_K)}[U_{\mathcal{P}}(\mathcal{S})] = 1/2$ by exchangeability (proof in §B.2), so $U_{\mathcal{P}}(\mathcal{S}) - 1/2$ is the gain over random search at the same budget.

Evaluations and statistics. Given a task $\tau$, let $\omega=(D,\hat{f},\xi)$ be the evaluation, with law $\mu_{\tau}$ on a space $\Omega$, and write $L(S;\omega)$ for the deployed loss equation 1. The deployment rule is deterministic given its inputs, so once $\omega$ is fixed the deployed loss of every subset is fixed: $L(S;\cdot)$ is a map on $\Omega$, which we assume measurable, as the expectation in equation 2 already requires. Write $\bar{P}=\mathbb{E}_{\tau\sim P}[\mu_{\tau}]$ and $\bar{Q}=\mathbb{E}_{\tau\sim Q}[\mu_{\tau}]$ for the marginal laws of $\omega$. A statistic is a measurable map $\Phi:\Omega\to\mathcal{X}$, and $P_{\Phi}=\bar{P}\circ\Phi^{-1}$, $Q_{\Phi}=\bar{Q}\circ\Phi^{-1}$ are the laws of $\Phi(\omega)$. In this notation, equation 2–equation 3 read

Upper bound on the improvement

Suppose we have access to the target distribution $P$ itself, so $U_{P}$ is known on all of $\Pi_{K}$ and selection carries no estimation error. We ask how many of the subsets in $\Pi_{K}$ improve upon the best subset seen so far. Suppose the $M=\binom{C}{K}$ subsets are revealed in uniformly random order. A subset improves the oracle’s choice precisely when its utility sets a new record.

The count is a sum of independent Bernoulli variables with mean $\log M+O(1)$ [40]: improvements arrive at rate $1/i$ and thin out as the search proceeds. Since $\log M=K\log(C/K)+O(K)$, at fixed $K\ll C$ doubling the pool adds about $K\log2$ of them. Meta-tuning thus is the problem of recovering enough of the $\log M$ improvements, having access only to the $N$ source tasks.

Transfer from synthetic to real tasks

The target distribution is not available for meta-tuning: we observe only a benchmark of real tasks, which we reserve for evaluation. We therefore select the subset on a source distribution $Q$ of synthetic tasks and transfer it. We sample $N$ independent source tasks $\tau_i\sim Q$ and one evaluation $(D_i,\hat{f}_i,\xi_i)$ for each. Write $L_i(S)=L(S;D_i,\hat{f}_i,\xi_i)$. The empirical counterpart of equation 3 is

and $\mathbb{E}[\widehat{U}_{N}(S)]=U_{Q}(S)$ for every fixed subset. Its expectation is $U_{Q}$, not $U_{P}$; we quantify the price of this approximation.

its suboptimality on the source tasks. Under Assumption 1, with probability at least $1-\delta$,

The bound separates selection error, source-target mismatch, and finite-task uncertainty. The number of source tasks reduces the second term, of order $\sqrt{\log M/N}$, but cannot remove systematic reversals of subset preferences ($\varepsilon$) or the error of the selection algorithm ($\eta_N$). We next investigate when Assumption 1 holds.

Hence what must transfer is the ranking of subsets by deployed loss. Under T this reduces to the validation and test ordering of individual configurations. Whether the synthetic prior preserves these orderings is an empirical question.

Analysis of the forward-selection heuristic

The inner expectation in equation (4) runs over $\binom{C}{K}$ references, so in practice we draw $R$ independent references $\mathcal{R}_r\sim\operatorname{Unif}(\Pi_K)$ and optimize

References thus add no bias, and their sampling noise vanishes as $1/R$.

Forward selection. Exhaustive search over $\binom{C}{K}$ subsets is impractical. Starting from $S_0=\varnothing$, we run

We return exactly $K$ configurations, even if an addition lowers the objective. The objective is not monotone in general, so forward selection is a heuristic with no generic greedy-optimality guarantee. In Proposition 2, both the gap between $\widetilde{U}_{N,R}$ and $\widehat{U}_N$ and the greedy suboptimality enter through $\eta_N$.

Proofs

Proof of the random-$K$ baseline. Let $\mathcal{S},\mathcal{R}$ be i.i.d. uniform on $\Pi_K$, independent of $(\tau,\omega)$, and write $a=L(\mathcal{S};\omega)$, $b=L(\mathcal{R};\omega)$, so that $\mathbb{E}_{\mathcal{S}}[U_{\mathcal{P}}(\mathcal{S})]=\mathbb{E}_{\tau,\omega}\mathbb{E}_{\mathcal{S},\mathcal{R}}[h(a,b)]$. For fixed $(\tau,\omega)$, the pairs $(\mathcal{S},\mathcal{R})$ and $(\mathcal{R},\mathcal{S})$ have the same law, so $\mathbb{E}_{\mathcal{S},\mathcal{R}}[h(a,b)]=\mathbb{E}_{\mathcal{S},\mathcal{R}}[h(b,a)]$, and adding the two with $h(a,b)+h(b,a)=1$ gives $\mathbb{E}_{\mathcal{S},\mathcal{R}}[h(a,b)]=\frac{1}{2}$ for every $(\tau,\omega)$, hence $\mathbb{E}_{\mathcal{S}}[U_{\mathcal{P}}(\mathcal{S})]=\frac{1}{2}$.

Proof of Proposition 1. Write $M=\binom{C}{K}$ and let $U_i=U_{\mathcal{P}}(\mathcal{S}_i)$ be the utility of the $i$th revealed subset. Let $\rho_i$ be the rank of $U_i$, from largest to smallest, among $U_1,\ldots,U_i$, so that $I_i=1_{\rho_i=1}$ indicates a record. We first show that $\rho_1,\ldots,\rho_M$ are independent with $\rho_i$ uniform on $\{1,\ldots,i\}$; this is the classical record theorem of Rényi [40].

Let $\pi$ be the permutation that sorts $(U_1,\ldots,U_M)$. Because the reveal order is uniform and the utilities are distinct, every ordering is equally likely, so $\pi$ is uniform over the $M!$ permutations. The rank vector $(\rho_1,\ldots,\rho_M)$ is a function of $\pi$, and this function is a bijection onto $\{1\}\times\{1,2\}\times\cdots\times\{1,\ldots,M\}$: it is injective because $\pi$ is recovered by inserting $U_i$ at position $\rho_i$ among $U_1,\ldots,U_{i-1}$ for $i=1,\ldots,M$, and both sets have $M!$ elements. Hence $(\rho_1,\ldots,\rho_M)$ is uniform on that product set, and for each marginal we can calculate

since the retained sums have $1\cdot2\cdots(i-1)\cdot(i+1)\cdots M=M!/i$ terms. For independence, we observe that $\Pr(\rho_1=k_1,\ldots,\rho_M=k_M)=1/M!=\Pr(\rho_1=k_1)\cdots\Pr(\rho_M=k_M)$. Therefore we also have that $\Pr(I_i=1)=\Pr(\rho_i=1)=1/i$, and $I_1,\ldots,I_M$ are independent because each is a function of a different $\rho_i$. By linearity, $\mathbb{E}\sum_i I_i=\sum_i\Pr(I_i=1)=H_M$, and comparing the sum with $\int x^{-1}dx$ gives $\log(M+1)\le H_M\le1+\log M$. Finally $\binom{C}{K}\le M\le\binom{C}{K}Ke^K$ gives $\log M=K\log(C/K)+O(K)$.

Proof of Proposition 2. Fix $\mathcal{S}\in\Pi_K$. By equation (4), $\widehat{U}_N(\mathcal{S})$ is the mean of $N$ i.i.d. terms in $[0,1]$ with mean $U_{\mathcal{Q}}(\mathcal{S})$. Hoeffding’s inequality gives $\Pr(|\widehat{U}_N(\mathcal{S})-U_{\mathcal{Q}}(\mathcal{S})|>t)\le2\exp(-2Nt^2)$, and a union bound over the $\binom{C}{K}$ subsets gives

Setting the right-hand side to $\delta$ and solving for $t$ yields $t_N=\sqrt{\log(2\binom{C}{K}/\delta)/(2N)}$. On the complementary event, which has probability at least $1-\delta$, let $\mathcal{S}^*$ maximize $U_{\mathcal{P}}$. Then

by equation (5)

by uniform concentration

by the definition of $\eta_N$.

The bound holds for any selector. Ours maximizes the Monte Carlo objective $\widetilde{U}_{N,R}$ rather than $\widehat{U}_N$, and both that gap and the greedy heuristic’s are part of $\eta_N$.

Proof of Lemma 1. Since $\mathcal{R}\sim\operatorname{Unif}(\Pi_K)$ is independent of $(\tau,\omega)$ and all terms are bounded, the objective equation (3) and the factoring equation (7) give

where the last equality uses that $\bar{P} = \mathbb{E}_{\tau\sim P}[\mu_{\tau}]$ is the marginal law of $\omega$ and $P_{\Phi}$ is the law of $\Phi(\omega)$ under it; likewise $U_Q(S) = \mathbb{E}_{R}\mathbb{E}_{Q_{\Phi}}[g_{S,R}]$. For any measurable $g : \mathcal{X} \to[0,1]$, the layer-cake formula gives $\lvert\mathbb{E}_{P}g - \mathbb{E}_{Q}g \rvert= \lvert\int_{0}^{1}(P(g > t) - Q(g > t))\,\mathrm{d}t \rvert\leq\operatorname{TV}(P,Q)$. Applying it to each $g_{S,R}$ and averaging over $R$ gives $\lvert U_P(S) - U_Q(S) \rvert\leq\operatorname{TV}(P_{\Phi},Q_{\Phi})$ for every $S$. Applying this to $S$ and $T$ and using the triangle inequality yields equation (5) with $\varepsilon= 2\operatorname{TV}(P_{\Phi},Q_{\Phi})$.

If $\Phi= \kappa\circ\Psi$, then equation (7) holds with $\Psi$ and $g_{S,R} \circ\kappa$ in place of $\Phi$ and $g_{S,R}$. Since $\{\Phi\in A\} = \{\Psi\in\kappa^{-1}(A)\}$, $\lvert P_{\Phi}(A) - Q_{\Phi}(A) \rvert= \lvert P_{\Psi}(\kappa^{-1}(A)) - Q_{\Psi}(\kappa^{-1}(A)) \rvert\leq\operatorname{TV}(P_{\Psi},Q_{\Psi})$ for every $A$. □

Proof of Proposition 3. Let $\Phi(\omega) = (\operatorname{rank}_{\omega}(S))_{S \in\Pi_K}$ be the vector of ranks of the losses $\{L(S;\omega)\}_{S \in\Pi_K}$, equal losses receiving equal rank; it takes values in a finite set. Since $h(a,b)$ depends only on whether $a < b$, $a = b$, or $a > b$, and dense ranks preserve this, $h(L(S;\omega),L(R;\omega)) = \mathbf{1}_{\operatorname{rank}_{\omega}(S)<\operatorname{rank}_{\omega}(R)} + \frac{1}{2}\mathbf{1}_{\operatorname{rank}_{\omega}(S)=\operatorname{rank}_{\omega}(R)}$, which is equation (7). Lemma 1 gives $\varepsilon= 2\operatorname{TV}(P_{\Phi},Q_{\Phi})$.

Under $\mathcal{E}_{T}$, let $\Psi(\omega)$ be the pair of dense-rank vectors of the pool $C$ under the validation losses and under the test losses. The deployed member of $S$ is its validation-rank minimizer, ties broken by a fixed rule, and the deployed losses of two subsets compare as the test ranks of their deployed members, so $\Phi= \kappa\circ\Psi$ for a fixed $\kappa$. The second part of Lemma 1 applies. □

Proof of Proposition 4. Fix $S$, hold the source tasks fixed, and set $Z_r = \frac{1}{N}\sum_{i=1}^{N} h(L_i(S),L_i(\mathcal{R}_r))$, so that $\widetilde{U}_{N,R}(S) = R^{-1}\sum_r Z_r$. The $Z_r$ are i.i.d. functions of the independent draws $\mathcal{R}_r$, lie in $[0,1]$, and have mean $\widehat{U}_N(S)$ by equation (4). Hence $\mathbb{E}_{R}[\widetilde{U}_{N,R}(S)] = \widehat{U}_N(S)$, and $\operatorname{Var}_{R}(\widetilde{U}_{N,R}(S)) = R^{-1}\operatorname{Var}(Z_1) \leq1/(4R)$, since a variable supported on $[0,1]$ has variance at most $1/4$. □