betaFeedback

WorksComputer Science

RAD-TFM: Robust and Domain-Adapted Tabular Foundation Models

Abstract

The development of tabular foundation models (TFMs) has accelerated in recent years, outperforming traditional ML methods on many tabular benchmarks. A key finding is that TFMs can be pretrained entirely on synthetic datasets, opening opportunities to design data generators that encourage desirable model properties. Prior work has mainly focused on crafting high-quality priors over generators to improve overall pretraining performance. Our key insight is that parameterizing the generator distribution enables an adversarial, distributional robustness perspective: during training, we can adapt the generator to emphasize datasets that are particularly challenging for the model, which we formalize by introducing an optimality gap measure. Further, we develop a method to align the synthetic data generation with real-world datasets from a given domain, constraining the adversary to generate realistic data. Together, these algorithms comprise the Robust and Domain-Adapted Tabular Foundation Models (RAD-TFM) pipeline, a model-agnostic adversarial training framework. Applied to the TabPFN V2 classifier, RAD-TFM improves performance across 6 diverse tabular benchmarks, with up to a 10.8 percentage-point improvement in mean normalized AUC over the strongest baseline, using fewer than 100k additional synthetic training datasets, less than 0.1% of the original pretraining data. Applying the same pipeline to TabICL V2 also improves performance across three major benchmarks, demonstrating the generality of the framework. These results highlight a promising new direction for targeted adversarial continued training of TFMs using synthetic data alone.

Introduction

Recent survey studies have challenged the deep learning (DL) community to improve upon traditional methods for learning from tabular data, such as boosted trees methods, having found that DL approaches still lagged behind in terms of performance [19]. In response, substantial progress has been made to close this gap, resulting in a variety of strong deep learning approaches being proposed over the past few years [35, 18, 20, 22, 7, 39, 42]. Among these new approaches, tabular foundation models (TFMs) which rely on in-context learning (ICL) [22, 31, 42] have emerged as a promising direction for classification and regression tasks with structured datasets. The benefits of this approach are two-fold. In many cases, the performance of the TFMs improves upon strong traditional baselines such as boosted trees [16, 14]. Additionally, this performance is achieved in a zero-shot framework, enabling high-quality predictions on new datasets in milliseconds when GPU-accelerated. Further, as we will explore in this work, since these TFMs are pretrained using synthetic data, there is significant opportunity for improvement as the data generation process itself is expanded and improved.

Pretraining TFMs relies on generating a large amount of diverse synthetic datasets [22]. As we will formally define in the following section, this generation process relies on constructing structural causal models (SCMs) from which datasets can be sampled [28, 34]. The structure of these SCMs is implicitly parameterized, giving significant control over the data generation process. All current publicly available, competitive TFMs [22, 31, 42] have been pretrained on datasets generated from a fixed prior distribution over these SCM parameters. However, fixed priors underrepresent certain regions of the parameter space, potentially degrading performance on real-world datasets with similar structure. This contributes to state-of-the-art TFMs still lagging behind tree-based methods on some benchmarks [25, 41]. In this work, we leverage the significant control provided by the data generation process to frame TFM training from an adversarial robustness perspective [24, 32]. Recent work has explored a narrower version of this idea, focusing on adjusting the weights of a specific class of SCMs in GAN-style training of TFMs [37]. In this work, we focus on a broad and highly flexible framework for adversarially robust training of TFMs. Going one step further, we ground this adversarial training process, constraining it to generating realistic data, yielding a robust and domain-adapted TFM.

Recent developments in TFMs, such as Real-TabPFN, have demonstrated the benefit of incorporating real-world data into the training process of TFMs [17]. However, as we will show in this work, finetuning on real datasets alone is often insufficient due to the lack of large-scale, high-quality tabular datasets, especially for more specific domains such as genomics or physical sciences. In this work, we will train TFMs on synthetic data, but we will constrain the data generation process to produce datasets that are similar to real-world datasets from a given domain. Previous work that aims to align a synthetic data generator with a tabular dataset typically take a GAN-based approach, focusing on constructing a generator to closely mimic a single real dataset [9, 27, 38]. In this work, we will expand beyond this one-to-one relationship, developing an algorithm that efficiently constructs a distribution over SCM-based tabular dataset generators that, in aggregate, align with real-world data across a large variety of meta-features capturing data schema, geometry, and learnability.

Our contributions can be summarized as follows: 1) We formalize distributionally-robust adversarial training over the SCM parameter space, allowing the model to adapt to challenging regions (e.g., varying numbers of features, categorical feature ratios, nonlinearities). We introduce an optimality gap objective and use it to target regions where the TFM underperforms relative to the best achievable performance. 2) We formalize the notion of domain-adapted TFMs in our DRO framework and introduce an efficient algorithm to construct a reference distribution over synthetic data generators that closely matches the real-data distribution for a given domain. 3) We propose an efficient, model-agnostic two-stage adversarial training algorithm for TFMs, which we call ROBUST AND DOMAIN-ADAPTED TABULAR FOUNDATION MODELS (RAD-TFM). 4) We apply RAD-TFM to TabPFN V2 [23], showing that with fewer than 100k additional training datasets, less than 0.1% of the original pretraining data, we can significantly improve its ranking on several real-world tabular benchmarks. We further demonstrate that the same training pipeline improves TabICL V2 [30], an architecturally distinct TFM. To organize this work, in Section 2, we first define the DRO formulation for robust and domain-adapted training of TFMs. Then, in Section 3, we introduce our method for finding an approximate reference distribution over data generators that aligns with real-world datasets. With these elements developed, in Section 4, we define our alternating two-stage algorithm for solving the core DRO problem. Finally, in Section 5, we apply our RAD-TFM pipeline and evaluate it on a variety of tabular benchmarks, both general and application-specific.

Problem Definition

We begin by defining some notation, and then introduce our theoretical formulation for RAD-TFM. Let Φ\Phi represent the space of hypothesis functions which define the relationship between features xx and outcome yy. The outcome yy can be discrete or continuous, but for this work, we focus on classification tasks. Then a hypothesis ϕ∈Φ\phi\in\Phi is a specific function which we can treat as a data generating mechanism. In the case of SCMs, Φ\Phi is the entire space of SCMs, and ϕ∈Φ\phi\in\Phi is a specific instance of an SCM, from which a dataset D∼p(ϕ)D \sim p(\phi) can be generated. In this work, we implement SCMs as randomly initialized multi-layer perceptrons (MLPs), as introduced in [22]. Full details of this process are provided in Appendix § 11. For example, we can sample the number of layers l∼p(l)l \sim p(l), the hidden size h∼p(h)h \sim p(h), the activation function a∼p(a)a \sim p(a), and the ratio of categorical features rcat∼p(rcat)r_{cat} \sim p(r_{cat}), among other dimensions. An SCM is generated by first sampling from the joint distribution over these hyperparameters, each of which is parameterized. For example, we might sample the ratio of categorical features rcat∼TruncNorm⁡(μrcat,a=0,b=1)r_{cat} \sim\operatorname{TruncNorm}(\mu_{r_{cat}},a=0,b=1) from the truncated normal distribution with some mean μrcat\mu_{r_{cat}} which parametrizes the distribution. Then, assuming we have similar parameterizations for the distribution over each hyperparameter, we define θ=(μl,μh,μrcat,…)\theta=(\mu_l,\mu_h,\mu_{r_{cat}},\ldots) as the parameterization of the joint distribution over all the hyperparameters. We define Θ\Theta as the space of the parameters, such that θ∈Θ\theta\in\Theta represents a specific parameterization. The generators are then sampled from ϕ∼p(Φ;θ)\phi\sim p(\Phi;\theta).

We now define the original objective for the Prior-Fitted Network (PFN) learning task and provide our adversarially robust formulation. Let gWg_{\mathbf{W}} be our predictive model parameterized by weight matrix W\mathbf{W}. As a PFN, gWg_{\mathbf{W}} takes as input a sequence of labeled training samples and unlabeled test samples, and predicts labels for the test samples. For a given data generator ϕ∈Φ\phi\in\Phi, the PFN loss is defined as the cross entropy loss over the test samples,

LPFN(W;ϕ)=E({xt,yt}∣D)∼p(ϕ)[−log⁡(gW(yt∣xt,D))].(1)\mathcal{L}_{PFN}(\mathbf{W};\phi)=\mathbb{E}_{(\{x_t,y_t\}\mid D)\sim p(\phi)}[-\log(g_{\mathbf{W}}(y_t\mid x_t,D))]. \tag*{(1)}

Since the model will be trained by sampling many data generators from the distribution ϕ∼p(Φ;θ)\phi\sim p(\Phi;\theta), we can write the original optimization problem from [22] as follows,

min⁡WEϕ∼p(Φ;θ)[LPFN(W;ϕ)].(2)\min_{\mathbf{W}}\mathbb{E}_{\phi\sim p(\Phi;\theta)}[\mathcal{L}_{PFN}(\mathbf{W};\phi)]. \tag*{(2)}

Since our goal is to learn a foundation model which performs well across all data generation schemes, we want to introduce an adversary who can shift the distribution over data generators to identify the regions over which gWg_{\mathbf{W}} yields the worst performance. However, there is a key subtlety in this formulation. Maximizing the loss alone could lead the maximizer to select data generators which produce datasets that are hard to learn for any model. Instead, we want to maximize the optimality gap between the model’s current performance and the best achievable performance. Since we are using the cross-entropy loss, we can frame this optimality gap in terms of the conditional entropy of the data. For any data generator ϕ\phi, let us denote the features as a random variable ZxZ_x and the target as ZyZ_y such that (Zx,Zy)∼p(ϕ)(Z_x,Z_y)\sim p(\phi). Then the Bayes’ optimal predictor is exactly f∗=p(Zy∣Zx)f^*=p(Z_y\mid Z_x) which achieves the cross-entropy loss LCE(f∗)=Hϕ(Zy∣Zx)L_{CE}(f^*)=H_{\phi}(Z_y\mid Z_x), the conditional entropy of the target ZyZ_y given the features ZxZ_x. It follows that for any fixed data generator, LPFN≥H(Zy∣Zx)\mathcal{L}_{PFN}\geq H(Z_y\mid Z_x), and LPFN−H(Zy∣Zx)≥0\mathcal{L}_{PFN}-H(Z_y\mid Z_x)\geq0 is the optimality gap. We can then state the adversarially robust optimization problem in terms of maximizing the optimality gap,

min⁡Wmax⁡θEϕ∼p(Φ;θ)[LPFN(W;ϕ)−Hϕ(Zy∣Zx)].(3)\min_{\mathbf{W}}\max_{\theta}\mathbb{E}_{\phi\sim p(\Phi;\theta)}[\mathcal{L}_{PFN}(\mathbf{W};\phi)-H_{\phi}(Z_y\mid Z_x)]. \tag*{(3)}

For brevity, we denote the objective function as δθ(W)=Eϕ∼p(Φ;θ)[LPFN(W;ϕ)−Hϕ(Zy∣Zx)]\delta_{\theta}(\mathbf{W})=\mathbb{E}_{\phi\sim p(\Phi;\theta)}[\mathcal{L}_{PFN}(\mathbf{W};\phi)-H_{\phi}(Z_y\mid Z_x)]. While Equation (3) is the precise theoretical optimization problem we wish to solve, in reality, estimating the conditional entropy Hϕ(Zy∣Zx)H_{\phi}(Z_y\mid Z_x) or evaluating it analytically is often infeasible. Therefore, rather than estimate Hϕ(Zy∣Zx)H_{\phi}(Z_y \mid Z_x) directly, in this work, we replace this term with the minimum cross-entropy loss over a collection of baseline estimators f1,…,fef_1,\ldots,f_e, such as XGBoost, Catboost, and Random Forests [8, 13, 6], which are fit to each generated dataset. Since min⁡i∈[e]LPFN(fi;ϕ)≥H(Zy∣Zx)\min_{i\in[e]} \mathcal{L}_{PFN}(f_i;\phi) \ge H(Z_y \mid Z_x), it follows that Eϕ∼p(Φ;θ)[LPFN(W;ϕ)−min⁡i∈[e]LPFN(fi;ϕ)]≤δθ(W)\mathbb{E}_{\phi\sim p(\Phi;\theta)}[\mathcal{L}_{PFN}(W;\phi)-\min_{i\in[e]}\mathcal{L}_{PFN}(f_i;\phi)] \le\delta_\theta(W), providing us with a lower bound on the optimality gap. We then modify our original objective function, defining

δ^θ(W)=Eϕ∼p(Φ;θ)[LPFN(W;ϕ)−min⁡i∈[e]LPFN(fi;ϕ)].(4)\widehat{\delta}_{\theta}(W)=\mathbb{E}_{\phi\sim p(\Phi;\theta)}\left[\mathcal{L}_{PFN}(W;\phi)-\min_{i\in[e]}\mathcal{L}_{PFN}(f_i;\phi)\right]. \tag*{(4)}

Since δ^θ(W)≤δθ(W)\widehat{\delta}_{\theta}(W)\le\delta_{\theta}(W), it follows that δ^θ(W)>0\widehat{\delta}_{\theta}(W)>0 implies δθ(W)>0\delta_{\theta}(W)>0, such that for any parameter estimated to have a positive optimality gap, the model gWg_W does indeed have room for improvement. Further, this lower bound can always be improved by adding more baseline estimators with varying strengths and inductive biases. In practice, maximizing over a single parameterization could lead to the model overfitting to a particular region of the parameter space. Further, for any given domain or application, we want to ensure that the generated synthetic datasets do not stray too far from the types of datasets observed in that domain. We therefore frame this problem from a distributionally robust optimization (DRO) perspective by constraining the actions of the maximizer [32]. Rather than a specific parameterization θ\theta, we allow the maximizer to select a distribution Q∈ΔΘQ\in\Delta_{\Theta} over parameters, where ΔΘ\Delta_{\Theta} is the simplex over Θ\Theta. We assume Θ\Theta is discrete, which is reasonable in practice since the parameters are either categorical or can be easily discretized (e.g. μrcat∈{0,0.1,…,1}\mu_{rcat}\in\{0,0.1,\ldots,1\}). To constrain the maximizer, we will assume access to a reference distribution P∈ΔΘP\in\Delta_{\Theta} that aligns with the data from a given domain. In Section 3, we will introduce our methodology for finding such a reference distribution given a corpus of real-world datasets from a specific domain. Then, with the reference distribution PP, we can constrain the KL-divergence between QQ and PP, leading to the following optimization problem,

min⁡Wmax⁡Q∈ΔΘEθ∼Q[δ^θ(W)]subject toDKL(Q∥P)≤ρ.(5)\begin{aligned} \min_W\quad&\max_{Q\in\Delta_{\Theta}} \quad\mathbb{E}_{\theta\sim Q}[\widehat{\delta}_{\theta}(W)]\\ &\text{subject to}\quad D_{KL}(Q\mathbin{\|}P)\le\rho. \tag*{(5)} \end{aligned}

As we prove in Appendix § 10, the optimal distribution for the maximizer in Equation (5) is a soft-max distribution. Specifically, denoting Q∗=(q1∗,…,q∣Θ∣∗)Q^*=(q_1^*,\ldots,q_{|\Theta|}^*) as the optimal distribution and P=(p1,…,p∣Θ∣)P=(p_1,\ldots,p_{|\Theta|}) as the reference distribution, for any i∈∣Θ∣i\in|\Theta|, qi∗∝piexp⁡(η⋅δθi(W))q_i^*\propto p_i\exp(\eta\cdot\delta_{\theta_i}(W)). The temperature η=η(ρ,δθ(W),P)\eta=\eta(\rho,\delta_{\theta}(W),P) is uniquely determined by the maximum KL-divergence, the optimality gaps, and the reference distribution. The DRO problem can then be rewritten as

min⁡W∑i=1∣Θ∣piexp⁡(η⋅δ^θi(W))∑j∈∣Θ∣pjexp⁡(η⋅δ^θj(W))⋅δ^θi(W).(6)\min_W \sum_{i=1}^{|\Theta|}\frac{p_i\exp(\eta\cdot\widehat{\delta}_{\theta_i}(W))}{\sum_{j\in|\Theta|}p_j\exp(\eta\cdot\widehat{\delta}_{\theta_j}(W))}\cdot\widehat{\delta}_{\theta_i}(W). \tag*{(6)}

In the following sections, we introduce our RAD-TFM approach to solving Equation (6). In Section 3, we develop our key contribution to domain adaptation for TFMs, learning the reference distribution PP over data generator parameters to align the generated data with real-world data. Then in Section 4, we incorporate the learned reference distribution into our DRO formulation and develop our method for solving this optimization problem using an iterative, two-stage approach.

Domain-Adapted Tabular Foundation Models

In our RAD-TFM methodology, the key to adapting a TFM to a specific domain is successfully constructing a reference distribution P∈ΔΘP\in\Delta_{\Theta} that connects a corpus of real-world datasets to a corresponding distribution over dataset generators. We will then use this reference distribution in our DRO formulation given in Equation (5) to adversarially generate datasets that challenge the TFM while remaining similar to the real-world datasets observed in the given domain. However, tabular datasets are structurally heterogeneous, with varying numbers and types of features, patterns, and targets. This motivates the need for a shared latent space for all tabular datasets that encodes semantic meaning through which we can project and compare datasets. To accomplish this, we rely on meta-feature extraction, a well-studied topic in meta-learning [5], to efficiently summarize any given tabular dataset into a fixed-size vector of statistical and information-theoretic properties. An illustration of our domain-adaptation scheme is given in Figure 2. Full details are given in Appendix § 8.

Domain adaptation for tabular foundation models

Figure 2. We perform domain adaptation for TFMs (e.g. healthcare, finance) by learning a mixture distribution over SCM generators that covers the distribution of real-world datasets from the given domain. We use a collection of meta-features capturing dataset schema, geometry, and learnability to encode and measure distance between datasets.

Following the definition of meta-features from [29] and [33], for a given dataset (X,y)∈S(X,y)\in\mathcal{S}, where S=⋃n≤N,d≤DRn×d\mathcal{S}=\bigcup_{n\le N,d\le D}\mathbb{R}^{n\times d} represents the space of tabular datasets up to some maximum size N×DN\times D, we can construct a characterization measure m:S→⋃dRdm : \mathcal{S} \to\bigcup_d \mathbb{R}^d, which produces a variable number of values depending on the shape of the dataset. For example, mm might extract the skewness of each numeric feature in the dataset separately. We can then introduce a summarization function σ:⋃dRd→Rk\sigma: \bigcup_d \mathbb{R}^d \to\mathbb{R}^k which condenses the characterization measure (e.g. by taking averages) into a fixed-size summarization vector of dimension kk. Combining these functions, we define the tabular dataset encoder as f((X,y))=σ(m((X,y)))f((X,y)) = \sigma(m((X,y))), projecting any tabular dataset into a semantically meaningful latent space. We provide details of the exact meta-features used in this work in Appendix A. Using this encoder, we can reason about the relationship between real datasets and synthetic datasets generated from the class of SCMs.

Suppose we have a corpus Dr={(Xi,yi)}i=1nr∈SnrD_r = \{(X_i,y_i)\}_{i=1}^{n_r} \in\mathcal{S}^{n_r} of real datasets from a given domain. At a high level, our algorithm works as follows: (1) encode each real dataset in the corpus as f((Xi,yi))=eif((X_i,y_i)) = e_i using our meta-feature extractor; (2) use Bayesian Optimization (BO) to find distributions over synthetic data generators such that the embeddings of the synthetic datasets provide partial coverage of the real dataset embeddings; (3) Aggregate the results to form a global distribution over data generators that minimizes the W1W_1-distance from the distribution of real datasets in the latent space. By creating a mixture over synthetic data distributions that each provide partial coverage of the real datasets, we can construct a reference distribution PP that aligns with an entire corpus of real datasets.

We now formalize the key steps in our DA algorithm where we align the distributions of real and synthetic datasets. We provide additional details in Appendix A. For each θ∈Θ\theta\in\Theta we have a distribution ϕ∼p(Φ;θ)\phi\sim p(\Phi;\theta) over data generators and, using the dataset encoder, a distribution e(θ)∼p(f(ϕ);θ)e(\theta) \sim p(f(\phi);\theta) over dataset embeddings, where f(ϕ)f(\phi) is short for the embedding of (X,y)∼p(ϕ)(X,y) \sim p(\phi), the dataset sampled from the specific generator. If we assume ∣Θ∣=M|\Theta| = M and denote Θ={θ1,…,θM}\Theta= \{\theta_1,\ldots,\theta_M\}, we can define Q(θi)=p(f(ϕ);θi)Q(\theta_i) = p(f(\phi);\theta_i) as the distribution over datasets in the latent space induced by θi\theta_i. Given that the embeddings of the corpus of real datasets is sampled from some distribution PrP_r, our goal is to solve the following optimization problem,

P∗∈arg⁡min⁡P∈ΔΘ  W1(Pr,∑i=1Mpi⋅Q(θi)),(7)P^* \in\underset{P \in\Delta_\Theta}{\arg\min}\; W_1\left(P_r,\sum_{i=1}^{M} p_i \cdot Q(\theta_i)\right), \tag*{(7)}

finding the mixture over these parameterized generator distributions that minimizes the 1-Wasserstein distance from the real data distribution. Solving this optimization problem recovers the reference distribution P∗P^* with the best possible coverage of the underlying domain-specific data distribution. In practice, we instead optimize the entropy-regularized optimal transport distance, which can be efficiently solved via the Sinkhorn Algorithm [11]. Since MM is large for any reasonable parameter search space, we approximate this search using Bayesian Optimization [15] with a surrogate objective designed to reward partial coverage of PrP_r. The resulting candidate distributions Q(θi)Q(\theta_i) are then combined by solving the mixture optimization problem in Equation (7), which has the side effect of pruning redundant and unhelpful generator distributions. The details of our method are given in Algorithm 1.

Robust Tabular Foundation Models

In this section, we give an overview of our two-stage optimization algorithm, ROBUST AND DOMAIN-ADAPTED TABULAR FOUNDATION MODELS (RAD-TFM), which is visualized in Figure 1. The pseudocode and additional details are provided in Appendix § 9. We first decompose the problem naturally into a maximization stage (parameter search) and a minimization stage (model training). We assume access to a pretrained TFM, gWg_{W}, with weights WW which will be used as the input model for our RAD-TFM algorithm. While not absolutely necessary, starting with a pretrained model gives us a warm-start for the model weights. In addition, we specify a fixed set of baseline models f1,…,fef_{1}, \ldots, f_{e} which will be used to estimate the optimality gap. We begin with the maximization stage, during which we freeze gWg_{W} to maximize the optimality gap in its current state. Since the cardinality of the parameter space is large in practice, we use a black-box optimization algorithm [1, 36, 3] to efficiently search the space for parameters with large optimality gaps.

Overview of RAD-TFM

Figure 1. Overview of RAD-TFM. We solve the core distributionally robust optimization problem by alternating between two stages: (a) searching over the SCM parameter space, we find synthetic data distributions with large estimated optimality gaps, and (b) sampling parameters proportional to their optimality gap, we generate synthetic datasets and continue training the TFM.

Using the black-box optimization algorithm, parameters θi\theta_{i} are proposed in sequence. For each θi\theta_{i} proposed, we estimate the optimality gap by sampling a fixed number, ndsn_{ds}, of generators ϕ1,…,ϕnds∼p(Φ;θi)\phi_{1}, \ldots, \phi_{n_{ds}} \sim p(\Phi; \theta_{i}) and datasets dj∼p(ϕj)d_{j} \sim p(\phi_{j}) and compute δ^θi=1nds∑j=1ndsLPFN(W;dj)−min⁡k∈[e]LPFN(fk,dj)\widehat{\delta}_{\theta_{i}} = \frac{1}{n_{ds}}\sum_{j=1}^{n_{ds}} \mathcal{L}_{PFN}(W; d_{j}) - \min_{k\in[e]} \mathcal{L}_{PFN}(f_{k}, d_{j}). Then for each proposed θi\theta_{i}, we must fit nds⋅en_{ds} \cdot e baseline estimators. However, each pair of dataset and baseline estimator can be fit independently, such that this step can be trivially parallelized, where each baseline estimator is fit to each dataset in parallel. In practice, we found that for nds=20n_{ds} = 20 and e=7e = 7, the estimated optimality gap δ^θi\widehat{\delta}_{\theta_{i}} could be computed in a matter of seconds when parallelized, given sufficient CPU cores (we used ncores=nds⋅en_{cores} = n_{ds} \cdot e). We repeat this process for a fixed number of trials, ntrialsn_{trials}, with the underlying optimization algorithm proposing new parameters θi\theta_{i} based on the observed estimated optimality gaps. At the end of this stage, we have a collection {(θi,δ^θi)}i=1ntrials\{(\theta_{i}, \widehat{\delta}_{\theta_{i}})\}_{i=1}^{n_{trials}} of parameters and associated estimated optimality gaps.

In the minimization stage, our goal is to train gWg_{W} on the regions of the parameter space with large optimality gaps. Following our derivation in Equation (6), we define a distribution QQ over Θ\Theta such that qi∝piexp⁡(η⋅δ^i(W))q_{i} \propto p_{i}\exp(\eta\cdot\widehat{\delta}_{i}(W)). The function η↦DKL(Q(η)∥P)\eta\mapsto D_{KL}(Q(\eta)\Vert P) is strictly increasing on (0,∞)(0,\infty), so we can quickly find η\eta given a fixed ρ\rho via one-dimensional root finding (e.g., bisection). Using the observations from the maximization stage, our optimization problem becomes

min⁡W∑i=1ntrialspiexp⁡(η⋅δ^θi(W))∑i∈[ntrials]piexp⁡(η⋅δ^θi(W))⋅δ^θi(W).(8)\min_{W}\sum_{i=1}^{n_{trials}}\frac{p_{i}\exp(\eta\cdot\widehat{\delta}_{\theta_{i}}(W))}{\sum_{i\in[n_{trials}]}p_{i}\exp(\eta\cdot\widehat{\delta}_{\theta_{i}}(W))}\cdot\widehat{\delta}_{\theta_{i}}(W). \tag*{(8)}

Then, to train the TFM, we generate each dataset within a batch by sampling θi∼Q\theta_{i}\sim Q and generator ϕ∼p(Φ;θi)\phi\sim p(\Phi;\theta_{i}). This sampling and dataset generation process is repeated for each batch during model training such that, in expectation, the loss is weighted according to Equation (8). After training gWg_{W} for nitern_{iter} iterations, we freeze the model return to the maximization stage, using the updated model weights to generate the next set of estimated optimality gaps. This max-min iteration is repeated until the objective in Equation (8) converges. In addition to the baseline estimators f1,…,fef_{1},\ldots,f_{e}, after a fixed number of max-min iterations, we incorporate the TFM with its original weights as an additional baseline model. This can help mitigate any unlearning that has occurred, encouraging the model to both improve upon strong baselines as well as its own performance at the start of training.

Experiments

Experiment Setup. We evaluate our RAD-TFM algorithm on broad benchmark classification datasets from TabArena [14], TabPertNet [40], and PMLB [26], as well as three domain-specific benchmarks derived from datasets available on OpenML [4]. We use the open-source model weights from TabPFN V2 [23] to initialize our TFM, and further apply the same pipeline without architectural modifications to the TabICL V2 classifier [30]. We provide a full description of the experiment setup and results in Appendix § 14. For the RAD-TFM algorithm, we use ntrials=100n_{\mathrm{trials}} = 100 and nds=20n_{\mathrm{ds}} = 20 datasets to estimate the optimality gap for each trial. During the minimization stage, we use a learning rate of lr=1e−5lr = 1\mathrm{e}{-5} and a batch size of b=64b = 64. We set ρ=0.1⋅ρmax⁡\rho= 0.1 \cdot\rho_{\max}, where ρmax⁡\rho_{\max} is the maximum KL-divergence given the learned reference distribution PP. Depending on the task, we train on 1,000–3,000 generated datasets per epoch and run the full max-min loop for e∈[5,30]e \in[5, 30] epochs, using at most 90k generated training datasets in any run. For baseline models, we use Random Forest, CatBoost, XGBoost, Logistic Regression, and MLPs. For the primary TabPFN benchmark tables, we also finetune TabPFN-V2 on each evaluation dataset for 10 epochs and report held-out performance as TabPFN (FT). In a separate compute-controlled comparison, we match fine-tuning to RAD-TFM by training-dataset exposure, batch size, and number of optimizer updates, evaluating checkpoints every 500 training datasets (Appendix § 15). We evaluate the conventional baseline algorithms using their respective default settings, and report TFM results without ensembling. All experiments were run on a single node with one A100 GPU; further details of the compute environment and SCM parameter space are provided in Appendix § 14 and D. Following previous work, we report normalized ROC-AUC as our primary metric, which scales raw ROC-AUC scores to [0,1], with the worst-performing model receiving a score of 0, and the best a score of 1 [21, 17]. This provides consistency when comparing performance across datasets, and measures how close each model comes to the best possible model on average. For multi-class tasks, we use the One-versus-One ROC-AUC score. We also report the mean rank and number of rank-1 wins for each model, based on the normalized AUC scores.

In Table 1, we report the main results from our experiments on the TabArena (N=21N = 21), TabPertNet (N=70N = 70), and PMLB (N=91N = 91) benchmarks. Across our three primary metrics, we observe that RAD-TFM improves performance over all baselines, including the original TabPFN-V2 and the

BenchmarkMetricLRMLPRFCatBoostXGBTabPFNn.e._{\mathrm{n.e.}}TabPFNn.e._{\mathrm{n.e.}} (FT)TabPFNn.e._{\mathrm{n.e.}} (RAD-TFM)
TabPertNet[40]Mean rank AUC5.46.04.64.25.23.73.93.1
TabPertNet[40]Mean Norm. AUC0.46840.36270.66460.74040.57430.73060.72990.8486{sup}*`
TabPertNet[40]Rank-1 Wins657679213
TabArena[14]Mean rank AUC5.77.25.54.05.23.62.72.1
TabArena[14]Mean Norm. AUC0.50700.25130.68700.82140.72930.88290.92150.9421{sup}*`
TabArena[14]Rank-1 Wins20013149
PMLB[26]Mean rank AUC5.87.04.54.34.73.33.33.1
PMLB[26]Mean Norm. AUC0.44970.27820.75800.77980.70360.88890.88940.9070
PMLB[26]Rank-1 Wins8210879416

Table 1. Experiment results across three tabular dataset benchmarks. For consistency, we report normalized AUC, with values scaled to [0,1] for each dataset [25, 23]. For rank-1 wins, we do not count ties, only outright wins.

BenchmarkMetricLRMLPRFCatBoostXGBTabPFNn.e._{\mathrm{n.e.}}TabPFNn.e._{\mathrm{n.e.}} (FT)TabPFNn.e._{\mathrm{n.e.}} (RAD-TFM)
EngineeringMean rank AUC6.97.34.33.44.93.33.42.6
EngineeringMean Norm. AUC0.29440.39510.84050.89520.78050.90040.89980.9313{sup}*`
EngineeringRank-1 Wins00142007
GenomicsMean rank AUC7.25.94.83.44.93.53.32.7
GenomicsMean Norm. AUC0.14810.43090.74390.86110.69790.88010.88050.9088
GenomicsRank-1 Wins00100002
HealthcareMean rank AUC5.36.94.53.55.03.83.83.2
HealthcareMean Norm. AUC0.58310.20500.72450.84460.67190.86960.86990.8974
HealthcareRank-1 Wins20224107
BenchmarkMetricTabPFNn.e._{\mathrm{n.e.}}TabPFNn.e._{\mathrm{n.e.}} (FT)TabPFNn.e._{\mathrm{n.e.}} (RAD-TFM)
TabPertNet[40]#Wins11537
AUC Change (%)-−0.0012.061
#Leaps-118
Norm. AUC (Std ↓)0.3450.3430.215
TabArena[14]#Wins1613
AUC Change (%)-0.1660.462
#Leaps-37
Norm. AUC (Std ↓)0.2230.1950.178
PMLB[26]#Wins15837
AUC Change (%)-0.0160.332
#Leaps-18
Norm. AUC (Std ↓)0.2100.2090.188

Table 2. Experiment results across three domain-specific tabular dataset benchmarks from OpenML [4]. For consistency, we report normalized AUC, with values scaled to [0,1] for each dataset [25, 23]. For rank-1 wins, we do not count ties, only outright wins.

finetuned versions. To measure the significance of our results, we first perform the Friedman test for repeated samples on the median normalized AUC scores [12, 2]. For the TabPertNet datasets, the Friedman test yielded p=3.5×10−15p = 3.5 \times10^{-15}, for TabArena, the test yielded p=4.3×10−13p = 4.3 \times10^{-13}, and for PMLB, the test yielded p=5.2×10−43p = 5.2 \times10^{-43}. Having established significance for the benchmark performance varying by model class, we performed the Wilcoxon signed-rank test pair-wise, comparing the RTFM version of TabPFN to the other models [10]. In Tables 1 and 2, we add an asterisk to indicate results for which the all Wilcoxon signed-rank tests between RAD-TFM and the alternative models achieves p<0.05p < 0.05. For the main benchmarks reported in Table 1, we see that all results are significant for RAD-TFM for TabPertNet and TabArena. Further, for PMLB, the largest p-value obtained was p=0.06p = 0.06, between RAD-TFM and TabPFN-V2 (FT), with p<0.05p < 0.05 for all other pair-wise tests. Our results are therefore almost strictly statistically significant, with RAD-TFM achieving the best performance for each metric across a variety of challenging tabular benchmarks.

In Table 2, we report results from experiments applying RAD-TFM to domain-specific benchmarks in Engineering (N=20N = 20), Genomics (N=11N = 11), and Healthcare (N=31N = 31), derived from OpenML. To construct these benchmarks, we aggregate all binary and multi-task classification datasets on OpenML matching a given tag (e.g. Engineering), then filter to datasets with at least n=200n = 200 samples, no more than n=10kn = 10k samples, and at most d=500d = 500 features. These benchmarks can be highly domain-specific, and differ significantly from common benchmarks such as TabArena. Similar to the other benchmarks, applying RAD-TFM improves the overall performance of TabPFN-V2 across domains, improving mean rank, normalized AUC, and outright wins. Additionally, for these more specific domains, finetuning TabPFN does not exhibit significant benefits. Across the three domain-specific benchmarks, RAD-TFM achieves 16 rank-1 wins, compared with only one for the original TabPFN and none for the fine-tuned model. The Friedman test returned p=1.3×10−11p = 1.3 \times10^{-11}, p=1.3×10−5p = 1.3 \times10^{-5}, and p=4.1×10−14p = 4.1 \times10^{-14} for Healthcare, Genomics, and Engineering respectively. Although our approach moves TabPFN’s performance in the right direction for these domains, the Wilcoxon signed-rank tests indicate that RAD-TFM only has statistically significant (p<0.05p < 0.05) improvement in normalized AUC for the Engineering benchmark. This is due in part to the small Genomics benchmark (N=11N = 11). The Healthcare benchmark is particularly challenging, indicating that it may be necessary to expand either the SCM parameter space or the meta-features used to represent the datasets to enable RAD-TFM to properly target this domain. Overall, these results indicate the potential for RAD-TFM to go beyond finetuning TFMs for domain-specific applications, noticeably improving model performance across three diverse domains.

RAD-TFM outperforms Fine-Tuning. To further investigate the impact of finetuning versus our RAD-TFM approach, in tbl:3, we compare these two methods against the original TabPFN V2 model in isolation. Across the three benchmarks, we report the number of outright wins among the three TabPFN variants and the relative percentage change in mean raw AUC from the frozen model. Additionally, we report a metric we refer to as the number of leaps, defined as the number of datasets for which the adapted TabPFN surpasses at least one conventional baseline that the original TabPFN-V2 did not outperform. This metric captures how often the method not only improves upon TabPFN-V2 incrementally, but improves it enough to surpass alternative model classes. This metric is key to our analysis, given that a major motivation for this work is the observation that, although TFMs are competitive with traditional methods, they do not consistently outperform them. In Table 3, we can see that applying RAD-TFM not only significantly improves the average change in AUC over finetuning alone, but also dominates in the number of wins and, importantly, the number of leaps. In the most extreme case, TabPertNet, finetuning has almost no effect on performance, while applying RAD-TFM leads to 18 leaps, such that TabPFN would now outperform at least one alternative method on 25.7% of datasets in the benchmark. Additionally, we report the standard deviation of normalized AUC scores for each model, which measures the dispersion of performance across datasets. We observe that the robust training process of RAD-TFM reduces this cross-dataset variability.

Generalization Across TFM Backbones. To evaluate the generality of the proposed training framework, we apply RAD-TFM to the open-source TabICL V2 classifier [30], a TFM with a different architecture from TabPFN V2. We use the same domain-adaptation and robust training pipeline, without modifying the TabICL architecture, and compare against the frozen backbone and compute-matched fine-tuning on TabPertNet, TabArena, and PMLB. Table 4 summarizes normalized AUC, rank-1 wins, and mean rank across the three benchmarks. Both RAD-TFM variants achieve the highest mean normalized AUC among the evaluated methods, and TabICL with RAD-TFM improves over the original TabICL on all three benchmarks (Appendix § 15). The improvement over the frozen TabICL is significant under a one-sided Wilcoxon signed-rank test on per-dataset normalized AUC (p=0.032p = 0.032), whereas the corresponding improvement from fine-tuning is not (p=0.148p = 0.148). The direct comparison of TabICL with RAD-TFM against fine-tuning favors RAD-TFM in mean normalized AUC, but is not statistically significant (mean gap 0.018; 95% bootstrap confidence interval [−0.005,0.043][-0.005, 0.043]). These results demonstrate that the benefits of RAD-TFM are not specific to TabPFN.

ModelMean Norm. AUCRank-1 WinsMean Rank
TabICL V2 (RAD-TFM)0.868214.23
TabPFN V2 (RAD-TFM)0.860114.73
TabICL V2 (FT)0.850154.37
TabICL V20.837184.67
TabPFN V2 (FT)0.81025.37
TabPFN V20.80615.45
CatBoost0.74386.21

Table 4. Generalization of RAD-TFM to the TabICL V2 backbone across TabPertNet, TabArena, and PMLB. Normalized AUC and ranks are computed over the expanded comparison of 11 models (including five classical baselines); selected models are shown. Scores are pooled across 177 datasets with defined normalized AUC and use a common normalization across all evaluated models.

Isolating Domain Adaptation and Robustness. We further ablate the two components of RAD-TFM by training TabPFN V2 using only the domain-adapted reference distribution (DA-only), or only robust parameter selection (Robust-only). In Table 5, each component independently improves upon the frozen model, while the full approach achieves the highest mean AUC on five of the six benchmarks. Across 236 datasets, both DA-only and Robust-only improve normalized AUC significantly relative to the frozen TabPFN (p=10−4p = 10^{-4} and p=0.028p = 0.028, respectively, using one-sided paired Wilcoxon tests). The effects are approximately additive on average, although the relative contribution varies by domain: domain adaptation accounts for most of the improvement on PMLB, while the two components complement one another on Engineering. Additional analysis is provided in Appendix § 15.

BenchmarkBaseDA-onlyRobust-onlyRAD-TFM
PMLB0.90880.91210.91090.9119
TabPertNet0.80190.80930.81130.8185
TabArena0.87250.87380.87430.8765
Genomics0.96750.96990.96790.9713
Healthcare0.89150.89370.89380.8951
Engineering0.94020.94030.93860.9422
Pooled0.87800.88190.88180.8851

Table 5. Ablation of domain adaptation (DA) and robust SCM parameter selection, reporting mean test AUC for TabPFN V2 across six benchmarks. The full method combines both components.

Compute-Matched Fine-Tuning. To separate the benefit of the training distribution from training duration, we additionally train TabPFN V2 on real reference datasets with the same batch size and number of training datasets and optimizer updates as the corresponding RAD-TFM runs. Both approaches exhibit small gains at early checkpoints, but their trajectories diverge with additional training. Across TabArena, PMLB, and TabPertNet, fine-tuning eventually falls below the frozen backbone, while RAD-TFM continues to improve. On TabArena, for example, the selected RAD-

TFM checkpoint from a training run with 30k generated-dataset exposures yields a mean raw AUC gain of +0.0040 relative to the frozen model, compared with −0.0844 for fine-tuning evaluated at the 30k checkpoint. The full checkpoint comparisons are reported in Appendix § 15.

Learned SCM Parameters and Generator Coverage. The learned adversarial distributions vary substantially across benchmarks (Appendix § 12). For example, Genomics favors generators with many numerical features, whereas Healthcare favors small, predominantly categorical datasets with higher noise. The Engineering benchmark instead selects larger, dense generators with numerical features. These differences illustrate the importance of adapting both dataset-level properties and the underlying data-generating mechanisms. However, domain adaptation is necessarily limited by the expressiveness of the available SCM family. On a foreign-exchange benchmark with pronounced temporal structure, the original non-temporal generators provide poor coverage of the real-data distribution, and RAD-TFM does not improve upon the frozen TabPFN. Extending the generator family with a temporal SCM improves alignment and yields a modest gain in AUC. This case illustrates both a limitation of the current generator space and the potential of domain-specific synthetic generation (Appendix § 13).

Conclusion

We introduce RAD-TFM, a model-agnostic framework for robust, domain-adapted continued training of TFMs. We demonstrate its ability to improve TabPFN-V2 using less than 0.1% of the original pretraining data, across six different tabular benchmarks. We further improve the architecturally distinct TabICL V2 classifier across three major benchmarks, demonstrating the generality of our approach. We show that constraining the robust training process to generate synthetic data that is similar to real-world data can be done efficiently, and that both domain adaptation and robust parameter selection independently contribute to performance. Compute-matched comparisons further distinguish the gains from training on robust, domain-adapted synthetic data from simply training for longer on real datasets. The foreign-exchange experiments further highlight the role of the synthetic generator family: extending its support to temporal data improves domain coverage and enables further adaptation. Our framework can also be extended to other TFMs and regression tasks. The RAD-TFM algorithm is highly-parallelizable, enabling large-scale training and efficient parameter search. Together, these results demonstrate how robust optimization and flexible SCM-based synthetic data generation can improve TFM performance across diverse domains.

Acknowledgments and Disclosure of Funding

Funding. This work was supported by a research grant from the MIT–IBM Computing Research Lab.

Competing interests. The authors declare no competing interests.

Details for Domain-Adapted TFMs

In this Section, we provide a detailed overview of the domain-adaptation process introduced in Section 3. We begin by defining the set of meta-features used to encode tabular datasets, then define our surrogate reward for the Bayesian Optimization process to maximize partial coverage of the real data distribution, and finally introduce our algorithm for constructing the global mixture over synthetic data generators. Although the overall pipeline involves multiple steps, the meta-features we define are inexpensive to compute, and we are able to solve the global mixture optimization efficiently via the Sinkhorn algorithm. As a result, we found that for all of our experiments, we can recover the approximate reference distribution over synthetic data generators in under 10 minutes, using a single A100 GPU on standard hardware. Further, this process only needs to be run once for each domain, since the same reference distribution can be re-used across many training runs of the RAD-TFM algorithm. We summarize the pseudocode for our domain-adaptation pipline in Algorithms 1 and 2.

Meta-Feature Extraction. For a tabular dataset (X,y)(X, y) with nn samples and pp input features, we construct a meta-feature embedding

ϕ(X,y)=[ϕschema(X)ϕgeom(X)ϕlearn(X,y)]∈Rd.\phi(X, y) = \begin{bmatrix} \phi_{\mathrm{schema}}(X) \\ \phi_{\mathrm{geom}}(X) \\ \phi_{\mathrm{learn}}(X, y) \end{bmatrix} \in\mathbb{R}^{d}.

The representation is designed to separate three sources of variation: coarse dataset schema and marginals, affine-invariant geometric structure of the numeric covariates, and inexpensive proxies for supervised learnability. In the classification setting used in our experiments, the corresponding block dimensions are (dschema,dgeom,dlearn)=(24,24,16)(d_{\mathrm{schema}}, d_{\mathrm{geom}}, d_{\mathrm{learn}}) = (24, 24, 16).

Throughout, for any finite vector v=(v1,…,vm)v = (v_1, \ldots, v_m), we write

S(v)=(mean⁡(v), std⁡(v), Q0.25(v), median⁡(v), Q0.75(v))S(v) = \bigl(\operatorname{mean}(v),\ \operatorname{std}(v),\ Q_{0.25}(v),\ \operatorname{median}(v),\ Q_{0.75}(v)\bigr)

for the five-dimensional summary operator applied repeatedly below. The schema block contains the global descriptors: log⁡(1+p)\log(1 + p), categorical feature ratio, total missing fraction, together with the fraction of categorical features that are binary, and the summaries SS (per-feature missing rate), SS (numeric unique ratio), SS (categorical unique ratio), SS (categorical normalized entropy). Note that for this work, we do not encode schema information regarding the number of samples in the dataset, since we currently fix the number of samples generated for synthetic datasets to be a constant during model training. In future work, we plan to include these features of the dataset schema, but we exclude them for now for training stability purposes. These quantities preserve coarse structural properties of the dataset that should not be removed by downstream normalization.

Let Xnum∈Rn×pnumX_{\mathrm{num}} \in\mathbb{R}^{n \times p_{\mathrm{num}}} denote the numeric submatrix. For each numeric feature jj, we compute the mean μj\mu_j and standard deviation σj\sigma_j using observed entries only, impute missing values by μj\mu_j, and form the standardized column

X~ij=Xij−μjσj\widetilde{X}_{ij} = \frac{X_{ij} - \mu_j}{\sigma_j}

All geometric features are computed from X~num\widetilde{X}_{\mathrm{num}}, making this block invariant to per-feature affine transformations Xij↦ajXij+bjX_{ij} \mapsto a_jX_{ij}+b_j with aj>0a_j>0. The geometric block contains S(∣feature skewness∣)S(\lvert\text{feature skewness}\rvert), S(feature kurtosis)S(\text{feature kurtosis}), and S(feature outlier rate)S(\text{feature outlier rate}), where the outlier rate is the fraction of observed standardized entries satisfying ∣X~ij∣>3\lvert\widetilde{X}_{ij}\rvert>3. To capture dependence structure, we form the correlation matrix RR of the standardized numeric features and include

S({∣Rjk∣:j<k}),max⁡j<k∣Rjk∣,S(\{\lvert R_{jk}\rvert:j<k\}), \qquad\max_{j<k}\lvert R_{jk}\rvert,

together with three spectral summaries,

λ1∑ℓλℓ,∑ℓ=1min⁡(5,r)λℓ∑ℓλℓ,exp⁡(−∑ℓπℓlog⁡πℓ),\frac{\lambda_1}{\sum_{\ell}\lambda_{\ell}}, \qquad \frac{\sum_{\ell=1}^{\min(5,r)}\lambda_{\ell}}{\sum_{\ell}\lambda_{\ell}}, \qquad \exp\left(-\sum_{\ell}\pi_{\ell}\log\pi_{\ell}\right),

where λ1≥λ2≥⋯≥0\lambda_1\ge\lambda_2\ge\cdots\ge0 are the eigenvalues of RR, rr is the rank, and πℓ=λℓ/∑kλk\pi_{\ell}=\lambda_{\ell}/\sum_k\lambda_k. The final term is the entropy-based effective rank.

The learnability block augments these unsupervised statistics with inexpensive surrogates for task difficulty. Specifically, it includes #classes, H(Y)H(Y), max⁡cPr⁡(Y=c)\max_c\Pr(Y=c), min⁡cPr⁡(Y=c)\min_c\Pr(Y=c), where

H(Y)=−∑cPr⁡(Y=c)log⁡Pr⁡(Y=c).H(Y)=-\sum_c\Pr(Y=c)\log\Pr(Y=c).

It further includes the summaries S(ηj2)S(\eta_j^2) over numeric features, and S(Vj)S(V_j) over categorical features, where ηj2\eta_j^2 is the ANOVA effect size between numeric feature jj and the class label, and VjV_j is Cramér’s VV between categorical feature jj and the class label. Finally, we include two global proxies for separability: leave-one-out nearest-centroid accuracy, and the ratio

mean pairwise distance between class centroidsmean within-class distance.\frac{\text{mean pairwise distance between class centroids}}{\text{mean within-class distance}}.

Given a collection of reference datasets {(X(i),y(i))}i=1N\{(X^{(i)},y^{(i)})\}_{i=1}^{N}, we first compute raw embeddings

u(i)=ϕ(X(i),y(i))∈Rd.u^{(i)}=\phi(X^{(i)},y^{(i)})\in\mathbb{R}^{d}.

For each coordinate kk, we then estimate the reference-set mean and variance,

μk=1N∑i=1Nuk(i),sk2=1N∑i=1N(uk(i)−μk)2,\mu_k=\frac{1}{N}\sum_{i=1}^{N}u_k^{(i)}, \qquad s_k^2=\frac{1}{N}\sum_{i=1}^{N}\left(u_k^{(i)}-\mu_k\right)^2,

and standardize each embedding coordinate according to

zk(i)=uk(i)−μksk.z_k^{(i)}=\frac{u_k^{(i)}-\mu_k}{s_k}.

Any generated dataset is mapped into the same latent space using the same (μk,sk)(\mu_k,s_k) estimated from the reference collection. After this per-coordinate standardization, we apply an explicit block reweighting. Writing

z=[zschemazgeomzlearn],zb∈Rdb,z=\begin{bmatrix} z_{\mathrm{schema}}\\ z_{\mathrm{geom}}\\ z_{\mathrm{learn}} \end{bmatrix}, \qquad z_b\in\mathbb{R}^{d_b},

we define the final transformed representation by

z^b=wbdb zb,\widehat{z}_b=\sqrt{\frac{w_b}{d_b}}\,z_b,

where wb>0w_b>0 is a user-specified block weight. Distances are then evaluated in the transformed space:

∥z^−z^′∥22=∑bwbdb∥zb−zb′∥22.\lVert\widehat{z}-\widehat{z}'\rVert_2^2=\sum_b\frac{w_b}{d_b}\lVert z_b-z_b'\rVert_2^2.

This normalization prevents large blocks from dominating the geometry solely by virtue of dimensionality. If each standardized coordinate has variance approximately one and within-block correlations are not too strong, then

E[∥zb∥22]≈db,E[∥z^b∥22]=wbdbE[∥zb∥22]≈wb.\mathbb{E}\left[\lVert z_b\rVert_2^2\right]\approx d_b,\qquad \mathbb{E}\left[\lVert\widehat{z}_b\rVert_2^2\right]=\frac{w_b}{d_b}\mathbb{E}\left[\lVert z_b\rVert_2^2\right]\approx w_b.

Thus the expected squared contribution of block bb is controlled by wbw_b rather than by its raw dimension. In practice, we set wb=13w_b = \frac{1}{3} for each of the three categories, balancing the contribution of each to the distance metric.

Parameter Search for Partial Coverage

With our dataset encoder and distance metric defined, we now introduce our search method for finding parameterizations of synthetic data generators that align with real-world data from some reference corpus. Since this is an unconstrained black-box optimization problem over a discrete search space, we use Bayesian Optimization to find parameters that cover parts of the reference corpus well [15]. To accomplish this, we now define our surrogate objective function that rewards partial coverage of the embedding space spanned by the reference corpus. For a given reference corpus of datasets, let {xi}i=1n⊂Rd\{x_i\}_{i=1}^{n} \subset\mathbb{R}^{d} denote its standardized reference embeddings after the same preprocessing and block reweighting described in the previous section. A trial parameter vector θ\theta induces a set of synthetic datasets; in the implementation, we generate mm datasets from ϕ∼p(Φ;θ)\phi\sim p(\Phi;\theta), embed them, and obtain a cloud {xj(θ)}j=1m⊂Rd\{x_j(\theta)\}_{j=1}^{m} \subset\mathbb{R}^{d}. The search objective is then defined in terms of how well this cloud covers the reference embeddings.

For each reference point xix_i, the we compute a soft nearest-neighbor distance to the generated cloud,

di(θ)=−τlog⁡(1m∑j=1mexp⁡(−∥xi−xj(θ)∥2τ)),(9)d_i(\theta) = -\tau\log\left(\frac{1}{m}\sum_{j=1}^{m}\exp\left(-\frac{\lVert x_i-x_j(\theta)\rVert_2}{\tau}\right)\right), \tag*{(9)}

with temperature τ>0\tau> 0. Distances are Euclidean and τ=1\tau= 1 by default. As τ→0\tau\to0, di(θ)d_i(\theta) approaches min⁡j∥xi−xj(θ)∥2\min_j \lVert x_i-x_j(\theta)\rVert_2, while for larger τ\tau it becomes a smoother average over nearby generated embeddings. This gives a differentiable surrogate for pointwise coverage without requiring an exact minimum.

The scalar objective returned to the optimizer at iteration tt is a residual-weighted average of these pointwise distances,

ft(θ)=−∑i=1nri(t)di(θ),ri(t)≥0,∑i=1nri(t)=1.(10)f_t(\theta) = -\sum_{i=1}^{n}r_i^{(t)}d_i(\theta), \qquad r_i^{(t)} \ge0, \qquad\sum_{i=1}^{n}r_i^{(t)} = 1. \tag*{(10)}

Maximizing ftf_t is therefore equivalent to minimizing a weighted coverage loss. The weights ri(t)r_i^{(t)}, which we refer to as the residuals, capture which parts of the reference data remain poorly covered by the synthetic data generators found so far. To define these residual weights, we track the coverage distance ci(t)c_i^{(t)} for each reference point. After evaluating a new parameter vector θt\theta_t, the candidate distances di(θt)d_i(\theta_t) are merged with the existing coverage state by a hard minimum,

ci(t+1)={di(θt),t=0,min⁡(ci(t),di(θt)),t≥1.(11)c_i^{(t+1)} = \begin{cases} d_i(\theta_t), & t = 0,\\ \min(c_i^{(t)},d_i(\theta_t)), & t \ge1. \end{cases} \tag*{(11)}

Thus ci(t)c_i^{(t)} records the best coverage attained so far by any previously evaluated generator in that cluster. The residual weights for the next trial are then recomputed from the updated coverage distances. The update is the linear normalization

ri(t+1)=ci(t+1)∑ℓ=1ncℓ(t+1).(12)r_i^{(t+1)} = \frac{c_i^{(t+1)}}{\sum_{\ell=1}^{n}c_\ell^{(t+1)}}. \tag*{(12)}

Reference points that are already well covered have small ci(t+1)c_i^{(t+1)} and therefore receive little residual mass, whereas persistently under-covered points retain large weights. The initial residual state is uniform, ri(0)=1/nr_i^{(0)} = 1/n. Consequently, early trials try to improve average coverage of the whole space, while later trials are pushed toward the portions of embedding space that remain inadequately represented. This is why we refer to the score as a partial coverage surrogate, since a single generator is not required to match the entire reference corpus by itself. Instead, the sequential parameter search encourages successive generators to cover complementary regions of the target embedding distribution. In practice, as we can see in Figure 3, there are no generators that adequately cover the entire space spanned by the reference corpus. Instead, by finding generators that partially cover this space and combining them, we are able to sufficiently cover the reference corpus.

Three scatter plots showing global mixture PCA at steps 0, 50, and 100

Figure 3. An example of our domain-adaptation methodology applied to the engineering datasets from OpenML [4]. Each color corresponds to a different parameterization of the synthetic data generator distribution. The black marks represent the embeddings of the real datasets. We plot the first two principal components of the meta-features extracted for the real datasets.

Solving the Global Mixture Problem

After the parameter search from the previous section is performed, we obtain TT candidate generators. Generator tt is represented by a cloud of embedded synthetic datasets {xj(θt)}j=1mt⊂Rd\{x_j(\theta_t)\}_{j=1}^{m_t} \subset\mathbb{R}^d, while the target collection of real datasets is represented by reference embeddings {xi}i=1n⊂Rd\{x_i\}_{i=1}^{n} \subset\mathbb{R}^d. In practice, we set mt=mm_t = m as a constant for consistency across trials. We want to fit a global distribution over generators by choosing mixture weights α=(α1,…,αT)\alpha= (\alpha_1,\ldots,\alpha_T) on the simplex that minimizes the transport distance between this global distribution over synthetic data generators and the distribution over the reference datasets.

The reference embeddings define the empirical measure

μ=1n∑i=1nδ(xi),(13)\mu= \frac{1}{n}\sum_{i=1}^{n}\delta(x_i), \tag*{(13)}

and a candidate generator mixture induces

να=∑k=1Tαt(1mt∑j=1mtδ(xj(θt))).(14)\nu_\alpha= \sum_{k=1}^{T}\alpha_t\left(\frac{1}{m_t}\sum_{j=1}^{m_t}\delta(x_j(\theta_t))\right). \tag*{(14)}

Thus each synthetic table embedding produced by generator kk receives mass αt/mt\alpha_t/m_t. We compare μ\mu and να\nu_\alpha using squared Euclidean cost c(x,y)=∥x−y∥22c(x,y)=\lVert x-y\rVert_2^2 and the entropically regularized optimal transport objective

OT⁡ε(μ,να)=min⁡Π∈U(a,b(α))⟨Π,C⟩+ε∑i,jΠij(log⁡Πij−1),(15)\operatorname{OT}_{\varepsilon}(\mu,\nu_\alpha)=\min_{\Pi\in U(a,b(\alpha))}\langle\Pi,C\rangle+\varepsilon\sum_{i,j}\Pi_{ij}(\log\Pi_{ij}-1), \tag*{(15)}

where ε>0\varepsilon>0 is the regularization parameter, ai=1/na_i=1/n, b(α)b(\alpha) collects the per-sample masses αt/mt\alpha_t/m_t, and U(a,b)U(a,b) is the set of couplings with marginals aa and bb. The entropic term makes the problem strictly convex in the transport plan and enables efficient solution by Sinkhorn matrix scaling [11].

Rather than solving the transport problem from scratch by general-purpose linear programming, we use the Sinkhorn algorithm in the log domain. Writing Cij=∥xi−yj∥22C_{ij}=\lVert x_i-y_j\rVert_2^2, the dual potentials are updated as

fi←ε(log⁡ai−log⁡∑jexp⁡(−Cij+gjε)),(16)f_i \leftarrow\varepsilon\left(\log a_i-\log\sum_j\exp\left(\frac{-C_{ij}+g_j}{\varepsilon}\right)\right), \tag*{(16)}
gj←ε(log⁡bj−log⁡∑iexp⁡(−Cij+fiε)),(17)g_j \leftarrow\varepsilon\left(\log b_j-\log\sum_i\exp\left(\frac{-C_{ij}+f_i}{\varepsilon}\right)\right), \tag*{(17)}

which yields the transport plan Πij=exp⁡((−Cij+fi+gj)/ε)\Pi_{ij}=\exp((-C_{ij}+f_i+g_j)/\varepsilon). This log-domain form is numerically stable and gives an efficient inner solver for repeated OT evaluations.

We then solve the outer mixture problem by minimizing the regularized transport cost with respect to α\alpha:

min⁡α∈ΔT−1OT⁡ε(μ,να).(18)\min_{\alpha\in\Delta^{T-1}}\operatorname{OT}_{\varepsilon}(\mu,\nu_\alpha). \tag*{(18)}

In practice, we reparameterize α\alpha by unconstrained logits η∈RT\eta\in\mathbb{R}^{T} with αt=exp⁡(ηt)/∑ℓ=1Texp⁡(ηℓ)\alpha_t=\exp(\eta_t)/\sum_{\ell=1}^{T}\exp(\eta_\ell), and optimize η\eta by gradient descent. Each outer step constructs the sample-weight vector b(α)b(\alpha), calls the Sinkhorn solver to evaluate OT⁡ε(μ,να)\operatorname{OT}_{\varepsilon}(\mu,\nu_\alpha), and backpropagates through that solution to update the logits. In this way, Sinkhorn serves as the inner OT subroutine inside the global mixture optimization. The result is a set of mixture weights over generators whose aggregated embedding distribution best matches the reference embeddings under the entropically regularized transport geometry.

RAD-TFM Algorithm Details

We summarize ROBUST AND DOMAIN-ADAPTED TABULAR FOUNDATION MODELS in Algorithms 3 and 4. A description of the algorithm is given in Section 4. To begin, we perform the domain-adaptation search from given by Algorithm 1 to construct a reference distribution PP over SCM parameters that aligns the synthetic data with a given reference corpus of real data. We then alternate between a maximization stage, in which we search over SCM parameters to find those with large optimality gaps, as in Equation (5), and a minimization stage, where we sample generators from

Algorithm 1 Pseudocode for Domain-Adapted TFMs

Require: Reference datasets Dref={(X(i),y(i))}i=1N\mathcal{D}_{\mathrm{ref}} = \{(\mathbf{X}^{(i)}, y^{(i)})\}_{i=1}^{N}, parameter space Θ\Theta, trials TT, datasets per trial mm, soft-coverage temperature τ\tau, transport regularization ε\varepsilon, Sinkhorn iterations LL, outer iterations OO

  1. Compute reference embeddings {xi}i=1N\{\mathbf{x}_i\}_{i=1}^{N} from Dref\mathcal{D}_{\mathrm{ref}}

  2. Standardize and reweight the embedding coordinates using statistics estimated from Dref\mathcal{D}_{\mathrm{ref}}

  3. Initialize an empty list of candidate generators and generated embedding clouds

  4. Initialize residual weights uniformly as r(0)=1N\mathbf{r}^{(0)} = \frac{1}{N}

  5. {(θt,{xj(θt)}j=1m)}t=1T←DomainParamSearch⁡({xi}i=1N,r,Θ,m,τ,T)\{(\theta_t, \{\mathbf{x}_j(\theta_t)\}_{j=1}^{m})\}_{t=1}^{T} \leftarrow\operatorname{DomainParamSearch}(\{\mathbf{x}_i\}_{i=1}^{N}, \mathbf{r}, \Theta, m, \tau, T)

  6. Form the empirical reference measure μ\mu from {xi}i=1N\{\mathbf{x}_i\}_{i=1}^{N}

  7. For each candidate generator θt\theta_t, form the empirical measure supported on its generated cloud Yt=1m∑j=1mδ(xj(θt))\mathcal{Y}_t = \frac{1}{m}\sum_{j=1}^{m} \delta(\mathbf{x}_j(\theta_t))

  8. Define να=∑t=1TαtYt\nu_{\alpha} = \sum_{t=1}^{T} \alpha_t\mathcal{Y}_t as the global mixture over synthetic data generators

  9. Solve α⋆∈arg⁡min⁡α∈ΔM−1OT⁡ε(μ,να)\alpha^\star\in\mathop{\arg\min}_{\alpha\in\Delta^{M-1}} \operatorname{OT}_{\varepsilon}(\mu,\nu_{\alpha}) using gradient-based optimization over mixture logits, with Sinkhorn⁡(μ,να,ε,L)\operatorname{Sinkhorn}(\mu,\nu_{\alpha},\varepsilon,L) as the inner OT solver, for OO outer iterations

  10. return Global mixture over generator parameters α⋆\alpha^\star

Algorithm 2 DOMAINPARAMSEARCH

Require: Reference embeddings {xi}i=1N\{\mathbf{x}_i\}_{i=1}^{N}, initial residuals r(0)\mathbf{r}^{(0)}, parameter space Θ\Theta, datasets per trial mm, soft-coverage temperature τ\tau, trials TT

  1. Initialize cover distances c(0)=(+∞,…,+∞)\mathbf{c}^{(0)} = (+\infty,\ldots,+\infty)

  2. Initialize history H(0)=∅\mathcal{H}^{(0)} = \varnothing

  3. for t=1,…,Tt = 1,\ldots,T do

  4. θt←BayesOPT⁡(H(t−1),Θ)\theta_t \leftarrow\operatorname{BayesOPT}(\mathcal{H}^{(t-1)},\Theta)

  5. Generate mm synthetic datasets from θt\theta_t, compute their embeddings, and collect them as {xj(θt)}j=1m\{\mathbf{x}_j(\theta_t)\}_{j=1}^{m}

  6. Compute the pointwise soft coverage distances {di(θt)}i=1N\{d_i(\theta_t)\}_{i=1}^{N} as in Equation (9)

  7. Compute the partial coverage reward ft(θt)=−∑i=1Nri(t−1)di(θt)f_t(\theta_t) = -\sum_{i=1}^{N} r_i^{(t-1)}d_i(\theta_t)

  8. for i=1,…,Ni = 1,\ldots,N do

  9. ci(t)←min⁡{ci(t−1),di(θt)}c_i^{(t)} \leftarrow\min\{c_i^{(t-1)},d_i(\theta_t)\}

  10. end for

  11. for i=1,…,Ni = 1,\ldots,N do

  12. ri(t)←ci(t)/∑ℓ=1Ncℓ(t)r_i^{(t)} \leftarrow c_i^{(t)} / \sum_{\ell=1}^{N} c_{\ell}^{(t)}

  13. end for

  14. Update the BO history as H(t)=H(t−1)∪{(θt,ft(θt))}\mathcal{H}^{(t)} = \mathcal{H}^{(t-1)} \cup\{(\theta_t,f_t(\theta_t))\}

  15. end for

  16. return Parameters and generated dataset point clouds {(θt,{xj(θt)}j=1m)}t=1T\{(\theta_t,\{\mathbf{x}_j(\theta_t)\}_{j=1}^{m})\}_{t=1}^{T}

an adversarial distribution to continue training the TFM. In Algorithm 4, we reference an abstract function SuggestParameters which represents any black-box optimization algorithm. In this case, we use the Optuna python package [1] and the Tree-structed Parzen Estimator [36] algorithm, which belongs to the class of Bayesian Optimization algorithms [15]. At each iteration, this algorithm uses the information gained from the previous trials (the estimated optimality gaps) to propose a new parameterization, balancing exploration and exploitation to efficiently find parameters corresponding to large optimality gaps. To begin the minimization stage, we first construct the distribution Q∈ΔPQ \in\Delta_{\mathcal{P}}. Rather than constructing ρ\rho as a parameter itself, we introduce a scalar parameter c∈(0,1)c \in(0,1) and define ρ=c⋅ρmax⁡\rho= c \cdot\rho_{\max}, such that ρ\rho is a given as a fraction of the maximum possible KL-divergence of QQ given the reference distribution PP. With ρmax⁡\rho_{\max} specified, we can find η\eta such that DKL(Q(η)∥P)=ρmax⁡D_{\mathrm{KL}}(Q(\eta)\|P) = \rho_{\max} using one-dimensional root finding (e.g. bisection). Then we generate each dataset in a training batch by sampling a parameter θi∼Q\theta_i \sim Q and generator ϕ∼p(Φ;θi)\phi\sim p(\Phi;\theta_i). We train the model for a fixed number of iterations and then return to the maximization stage. In our implementation, after five epochs, we incorporate the original TabPFN model as another baseline model. This can potentially Algorithm 3 Robust and Domain-Adapted Tabular Foundation Models (RAD-TFM)

Require: Initial TFM weights WW, baseline estimators {f1,…,fe}\{f_1,\ldots,f_e\}, nen_e, nitern_{iter}, ntn_t, ndn_d, c∈(0,1)c \in(0,1), reference distribution PP from Algorithm 1, parameter space Θ\Theta

  1. {(θi,δi)}i=1ntrials←\{(\theta_i,\delta_i)\}_{i=1}^{n_{trials}} \leftarrow ParameterSearch(Θ,W,{f1,…,fe},nt,nd)(\Theta,W,\{f_1,\ldots,f_e\},n_t,n_d)

  2. ρ←c⋅ρmax⁡\rho\leftarrow c \cdot\rho_{\max}

  3. η←\eta\leftarrow RootFinding(ρ,P,δ)(\rho,P,\delta) ▷ Find correct temperature for distribution

  4. Construct distribution Qi∝piexp⁡(η⋅δi)Q_i \propto p_i\exp(\eta\cdot\delta_i)

  5. for e∈[ne]e \in[n_e] do

  6. &nbsp;&nbsp;&nbsp;&nbsp;for t∈[niter]t \in[n_{iter}] do

  7. &nbsp;&nbsp;&nbsp;&nbsp;θ1,…,θnb∼Q\theta_1,\ldots,\theta_{n_b}\sim Q ▷ Sample parameters

  8. &nbsp;&nbsp;&nbsp;&nbsp;ϕ1,…,ϕnb∼p(Φ;θ1),…,p(Φ;θnb)\phi_1,\ldots,\phi_{n_b}\sim p(\Phi;\theta_1),\ldots,p(\Phi;\theta_{n_b})

  9. &nbsp;&nbsp;&nbsp;&nbsp;{(xi,yi)}i=1nb∼p(ϕ1),…,p(ϕnb)\{(x^i,y^i)\}_{i=1}^{n_b}\sim p(\phi_1),\ldots,p(\phi_{n_b}) ▷ Sample synthetic datasets

  10. &nbsp;&nbsp;&nbsp;&nbsp;{xtraini,xtesti,ytraini,ytesti}i=1nb←\{x^i_{train},x^i_{test},y^i_{train},y^i_{test}\}_{i=1}^{n_b}\leftarrow TrainTestSplit({(xi,yi)}i=1nb)(\{(x^i,y^i)\}_{i=1}^{n_b})

  11. &nbsp;&nbsp;&nbsp;&nbsp;l(gW)←LPFN(gW,{xtraini,xtesti,ytraini,ytesti}i=1nb)l(g_W)\leftarrow\mathcal{L}_{PFN}(g_W,\{x^i_{train},x^i_{test},y^i_{train},y^i_{test}\}_{i=1}^{n_b})

  12. &nbsp;&nbsp;&nbsp;&nbsp;W←update⁡(W,l(gW))W\leftarrow\operatorname{update}(W,l(g_W)) ▷ Gradient step

  13. &nbsp;&nbsp;&nbsp;&nbsp;end for

  14. {(θi,δi)}i=1ntrials←\{(\theta_i,\delta_i)\}_{i=1}^{n_{trials}}\leftarrow ParameterSearch(W,{f1,…,fe},nt,nd)(W,\{f_1,\ldots,f_e\},n_t,n_d)

  15. η←\eta\leftarrow RootFinding(Hmin⁡,δ)(H_{\min},\delta)

  16. Construct distribution Qi∝piexp⁡(η⋅δi)Q_i\propto p_i\exp(\eta\cdot\delta_i)

  17. end for

  18. Return Robust TFM model gWg_W

address regions of the parameter space where the model has reduced its performance by focusing on improving in other regions.

The RAD-TFM algorithm is highly parallelizable, both in the dataset generation steps and the parameter search. For a fixed parameterization θ\theta, each dataset generator ϕ∼p(Φ;θ)\phi\sim p(\Phi;\theta) is sampled independently from the same distribution, such that all datasets generated for a given batch can be created concurrently. The ParameterSearch algorithm can be completely parallelized within each trial. In our implementation, for each trial, we generate a batch of datasets and then fit and evaluate each model and dataset pair concurrently. For example, in our experiments, we set nd=20n_d=20 and used e=7e=7 baseline models, resulting in nds∗e=140n_{ds}*e=140 separate processes distributed across all available cores. In practice, the domain-adaptation stage is lightweight, while the overall training cost scales approximately linearly with the number of max-min iterations; we provide benchmark-specific compute details in Appendix G.

Algorithm 4 ParameterSearch

Require: TFM weights WW, baseline estimators {f1,…,fe}\{f_1,\ldots,f_e\}, num. trials ntn_t, num. datasets ndn_d

  1. Dθ={}D_\theta=\{\}

  2. for i∈[nt]i\in[n_t] do

  3. &nbsp;&nbsp;&nbsp;&nbsp;θi←\theta_i\leftarrow SuggestParameters(Dθ)(D_\theta) ▷ Black-box optimizer proposes new parameter

  4. &nbsp;&nbsp;&nbsp;&nbsp;for j∈[nd]j\in[n_d] do

  5. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ϕj∼p(Φ;θi)\phi_j\sim p(\Phi;\theta_i)

  6. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;xtrainj,xtestj,ytrainj,ytestj∼p(ϕj)x^j_{train},x^j_{test},y^j_{train},y^j_{test}\sim p(\phi_j)

  7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lj(gW)←LPFN(gW,xtrainj,ytrainj,xtestj,ytestj)l_j(g_W)\leftarrow\mathcal{L}_{PFN}(g_W,x^j_{train},y^j_{train},x^j_{test},y^j_{test})

  8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lb←∞l_b\leftarrow\infty

  9. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for fk∈{f1,…,fe}f_k\in\{f_1,\ldots,f_e\} do

  10. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lj(fk)←LPFN(fk,xtrainj,ytrainj,xtestj,ytestj)l_j(f_k)\leftarrow\mathcal{L}_{PFN}(f_k,x^j_{train},y^j_{train},x^j_{test},y^j_{test})

  11. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lb←min⁡(lb,lj(fk))l_b\leftarrow\min(l_b,l_j(f_k))

  12. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end for

  13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;δij←lj(gW)−lb\delta_i^j\leftarrow l_j(g_W)-l_b ▷ Estimate optimality gap

  14. &nbsp;&nbsp;&nbsp;&nbsp;end for

  15. &nbsp;&nbsp;&nbsp;&nbsp;δi←1nd∑j=1ndδij\delta_i\leftarrow\frac{1}{n_d}\sum_{j=1}^{n_d}\delta_i^j

  16. &nbsp;&nbsp;&nbsp;&nbsp;Dθ←Dθ∪{(θi,δi)}D_\theta\leftarrow D_\theta\cup\{(\theta_i,\delta_i)\}

  17. end for

  18. Return Parameters and estimated optimality gaps DθD_\theta

Theoretical Properties

In this section, we provide the proof that the optimal distribution for the maximizer in equation 5 is a softmax distribution such that qi∗∝piexp⁡(η⋅δθi(W))q_i^* \propto p_i \exp(\eta\cdot\delta_{\theta_i}(\mathbf{W})). Further, we show that in the absence of a reference distribution, if we take PP to be uniform, our formulation can be written as an entropy-constrained DRO, where QQ is constrained to have a certain minimum entropy.

Proposition 1. Let δ=(δθ1(W),…,δθn(W))∈Rn\boldsymbol{\delta} = (\delta_{\theta_1}(\mathbf{W}), \ldots, \delta_{\theta_n}(\mathbf{W})) \in\mathbb{R}^n and let P=(p1,…,pn)∈ΔnP = (p_1, \ldots, p_n) \in\Delta_n satisfy pi>0p_i > 0 for all ii. Consider the distributionally robust inner problem

max⁡Q∈Δn∑i=1nqiδθi(W)s.t.DKL(Q∥P)≤ρ,\max_{Q\in\Delta_n} \sum_{i=1}^{n} q_i\delta_{\theta_i}(\mathbf{W}) \quad\text{s.t.}\quad D_{\mathrm{KL}}(Q\|P) \leq\rho,

Let A=arg⁡max⁡i∈[n]δθi(W)\mathcal{A} = \arg\max_{i\in[n]} \delta_{\theta_i}(\mathbf{W}), and P(A)=∑i∈ApiP(\mathcal{A}) = \sum_{i\in\mathcal{A}} p_i. Assume that δθi(W)\delta_{\theta_i}(\mathbf{W}) is not constant across ii and 0<ρ<−log⁡P(A)0 < \rho< -\log P(\mathcal{A}). Then the following properties hold.

(i) The optimizer Q∗Q^* is strictly positive, and there exists a unique temperature parameter η>0\eta> 0 such that

qi∗=piexp⁡(δθi(W)/η)∑j=1npjexp⁡(δθj(W)/η),i=1,…,n,q_i^* = \frac{p_i\exp(\delta_{\theta_i}(\mathbf{W})/\eta)}{\sum_{j=1}^{n}p_j\exp(\delta_{\theta_j}(\mathbf{W})/\eta)}, \qquad i=1,\ldots,n,

where η\eta is determined implicitly by DKL(Q∗∥P)=ρD_{\mathrm{KL}}(Q^*\|P)=\rho.

(ii) If PP is the uniform distribution on [n][n], then the KL constraint DKL(Q∥P)≤ρD_{\mathrm{KL}}(Q\|P)\leq\rho is equivalent to the entropy constraint H(Q)≥log⁡n−ρH(Q)\geq\log n-\rho. In this case, the optimizer in part (i) reduces to the softmax distribution

qi∗=exp⁡(δθi(W)/η)∑j=1nexp⁡(δθj(W)/η),i=1,…,n,q_i^* = \frac{\exp(\delta_{\theta_i}(\mathbf{W})/\eta)}{\sum_{j=1}^{n}\exp(\delta_{\theta_j}(\mathbf{W})/\eta)}, \qquad i=1,\ldots,n,

with η>0\eta>0 chosen so that H(Q∗)=log⁡n−ρH(Q^*)=\log n-\rho.

Proof. Part (i). For notational simplicity, write δi=δθi(W)\delta_i = \delta_{\theta_i}(\mathbf{W}). The problem can be rewritten as

min⁡Q∈Δn−∑i=1nqiδis.t.DKL(Q∥P)≤ρ.\min_{Q\in\Delta_n} -\sum_{i=1}^{n}q_i\delta_i \quad\text{s.t.}\quad D_{\mathrm{KL}}(Q\|P)\leq\rho.

Since P∈ΔnP\in\Delta_n satisfies pi>0p_i>0 for all ii and

DKL(P∥P)=0<ρ,D_{\mathrm{KL}}(P\|P)=0<\rho,

Slater’s condition holds. Therefore, the KKT conditions are necessary and sufficient. The Lagrangian is

L(Q,η,μ,γ)=−∑i=1nqiδi+η(∑i=1nqilog⁡qipi−ρ)+μ(∑i=1nqi−1)−∑i=1nγiqi,\mathcal{L}(Q,\eta,\mu,\gamma)=-\sum_{i=1}^{n}q_i\delta_i+\eta\left(\sum_{i=1}^{n}q_i\log\frac{q_i}{p_i}-\rho\right)+\mu\left(\sum_{i=1}^{n}q_i-1\right)-\sum_{i=1}^{n}\gamma_iq_i,

where η≥0\eta\geq0, μ∈R\mu\in\mathbb{R}, and γi≥0\gamma_i\geq0.

We first show that the KL constraint must be active. If the KL constraint were inactive, then η=0\eta=0. In that case, the solution would have to maximize the linear objective over the simplex without an active KL constraint, and hence would place all mass on the set

A=arg⁡max⁡i∈[n]δi.\mathcal{A}=\arg\max_{i\in[n]}\delta_i.

Among all distributions supported on A\mathcal{A}, the one with smallest KL divergence from PP is the conditional distribution

qi=piP(A)1{i∈A},q_i=\frac{p_i}{P(\mathcal{A})}\mathbf{1}\{i\in\mathcal{A}\},

whose KL divergence is

DKL(Q∥P)=∑i∈ApiP(A)log⁡pi/P(A)pi=−log⁡P(A).D_{\mathrm{KL}}(Q\|P)=\sum_{i\in\mathcal{A}}\frac{p_i}{P(\mathcal{A})}\log\frac{p_i/P(\mathcal{A})}{p_i}=-\log P(\mathcal{A}).

Thus, if ρ<−log⁡P(A)\rho< -\log P(A), no distribution supported entirely on AA is feasible. Therefore the unconstrained maximizer is infeasible, so the KL constraint must be active and η>0\eta> 0.

For an interior optimum, complementary slackness gives γi=0\gamma_i = 0. The stationarity condition is then

−δi+η(1+log⁡qipi)+μ=0.-\delta_i + \eta\left(1+\log\frac{q_i}{p_i}\right)+\mu=0.

Rearranging,

log⁡qipi=δi−μη−1.\log\frac{q_i}{p_i}=\frac{\delta_i-\mu}{\eta}-1.

Exponentiating gives

qi=piexp⁡(δiη)exp⁡(−μη−1).q_i=p_i\exp\left(\frac{\delta_i}{\eta}\right)\exp\left(-\frac{\mu}{\eta}-1\right).

Enforcing ∑iqi=1\sum_i q_i=1 yields

qi(η)=piexp⁡(δi/η)Z(η),Z(η)=∑j=1npjexp⁡(δj/η).q_i(\eta)=\frac{p_i\exp(\delta_i/\eta)}{Z(\eta)},\qquad Z(\eta)=\sum_{j=1}^{n}p_j\exp(\delta_j/\eta).

Since pi>0p_i>0 and η>0\eta>0, this distribution is strictly positive.

It remains to show that there is a unique η>0\eta>0 satisfying DKL(Q(η)∥P)=ρD_{\mathrm{KL}}(Q(\eta)\mathbin{\|}P)=\rho. Let E(η)=∑i=1nqi(η)δiE(\eta)=\sum_{i=1}^{n}q_i(\eta)\delta_i. Then, for the distribution Q(η)Q(\eta),

log⁡qi(η)pi=δiη−log⁡Z(η),\log\frac{q_i(\eta)}{p_i}=\frac{\delta_i}{\eta}-\log Z(\eta),

so

DKL(Q(η)∥P)=∑i=1nqi(η)(δiη−log⁡Z(η))=E(η)η−log⁡Z(η).D_{\mathrm{KL}}(Q(\eta)\mathbin{\|}P)=\sum_{i=1}^{n}q_i(\eta)\left(\frac{\delta_i}{\eta}-\log Z(\eta)\right)=\frac{E(\eta)}{\eta}-\log Z(\eta).

Next,

ddηlog⁡Z(η)=1Z(η)∑i=1npiexp⁡(δi/η)(−δiη2)=−E(η)η2.\frac{d}{d\eta}\log Z(\eta)=\frac{1}{Z(\eta)}\sum_{i=1}^{n}p_i\exp(\delta_i/\eta)\left(-\frac{\delta_i}{\eta^2}\right)=-\frac{E(\eta)}{\eta^2}.

And,

dEdη=−Var⁡Q(η)(δ)η2.\frac{dE}{d\eta}=-\frac{\operatorname{Var}_{Q(\eta)}(\delta)}{\eta^2}.

Therefore,

ddηDKL(Q(η)∥P)=1ηdEdη−E(η)η2−ddηlog⁡Z(η)=1ηdEdη−E(η)η2+E(η)η2=−Var⁡Q(η)(δ)η3.\begin{aligned} \frac{d}{d\eta}D_{\mathrm{KL}}(Q(\eta)\mathbin{\|}P) &=\frac{1}{\eta}\frac{dE}{d\eta}-\frac{E(\eta)}{\eta^2}-\frac{d}{d\eta}\log Z(\eta)\\ &=\frac{1}{\eta}\frac{dE}{d\eta}-\frac{E(\eta)}{\eta^2}+\frac{E(\eta)}{\eta^2}\\ &=-\frac{\operatorname{Var}_{Q(\eta)}(\delta)}{\eta^3}. \end{aligned}

Since δi\delta_i is not constant across ii and Q(η)Q(\eta) is strictly positive, we have

Var⁡Q(η)(δ)>0\operatorname{Var}_{Q(\eta)}(\delta)>0

for every η>0\eta>0. We can then conclude that η↦DKL(Q(η)∥P)\eta\mapsto D_{\mathrm{KL}}(Q(\eta)\mathbin{\|}P) is strictly decreasing on (0,∞)(0,\infty).

Further, we have lim⁡η→∞Q(η)=P\lim_{\eta\to\infty}Q(\eta)=P, so lim⁡η→∞DKL(Q(η)∥P)=0\lim_{\eta\to\infty}D_{\mathrm{KL}}(Q(\eta)\mathbin{\|}P)=0. As η↓0\eta\downarrow0, Q(η)Q(\eta) converges to PP conditioned on the set of maximizers AA, namely

qi(0)=piP(A)1{i∈A},q_i(0)=\frac{p_i}{P(A)}\mathbf{1}\{i\in A\},

and therefore

lim⁡η↓0DKL(Q(η)∥P)=−log⁡P(A).\lim_{\eta\downarrow0}D_{\mathrm{KL}}(Q(\eta)\mathbin{\|}P)=-\log P(A).

Thus, for every 0<ρ<−log⁡P(A)0 < \rho< -\log P(A) there exists a unique η>0\eta> 0 such that DKL(Q(η)∥P)=ρD_{\mathrm{KL}}(Q(\eta)\|P) = \rho. By the KKT conditions, this Q(η)Q(\eta) is the global optimizer.

Part (ii). Suppose PP is uniform, so pi=1/np_i = 1/n for all ii. Then

DKL(Q∥P)=∑i=1nqilog⁡qi1/n=∑i=1nqilog⁡qi+∑i=1nqilog⁡n=∑i=1nqilog⁡qi+log⁡n=log⁡n−H(Q).\begin{aligned} D_{\mathrm{KL}}(Q\|P) &= \sum_{i=1}^{n} q_i \log\frac{q_i}{1/n} \\ &= \sum_{i=1}^{n} q_i \log q_i + \sum_{i=1}^{n} q_i \log n \\ &= \sum_{i=1}^{n} q_i \log q_i + \log n \\ &= \log n - H(Q). \end{aligned}

Therefore, DKL(Q∥P)≤ρD_{\mathrm{KL}}(Q\|P) \leq\rho is equivalent to log⁡n−H(Q)≤ρ\log n - H(Q) \leq\rho, or H(Q)≥log⁡n−ρH(Q) \geq\log n - \rho. Finally, substituting pi=1/np_i = 1/n into the solution from part (i) gives

qi∗=(1/n)exp⁡(δi/η)∑j=1n(1/n)exp⁡(δj/η)=exp⁡(δi/η)∑j=1nexp⁡(δj/η).q_i^* = \frac{(1/n)\exp(\delta_i/\eta)}{\sum_{j=1}^{n}(1/n)\exp(\delta_j/\eta)} = \frac{\exp(\delta_i/\eta)}{\sum_{j=1}^{n}\exp(\delta_j/\eta)}.

with η>0\eta> 0 chosen so that DKL(Q∗∥P)=ρD_{\mathrm{KL}}(Q^*\|P) = \rho. Using DKL(Q∗∥P)=log⁡n−H(Q∗)D_{\mathrm{KL}}(Q^*\|P) = \log n - H(Q^*), this is equivalently H(Q∗)=log⁡n−ρH(Q^*) = \log n - \rho. This proves the claim.

SCM Parameters

We summarize the MLP-based SCM parameter space used in our experiments in Table 6. In this work, we instantiate SCMs as randomized MLPs, so the search space includes both architectural parameters and dataset-level properties. The MLP-specific parameters are the hidden width μh\mu_h, number of layers μl\mu_l, input width μxin\mu_{x_{\mathrm{in}}}, activation function μa\mu_a, shuffled categorical ratio μpshuffle\mu_{p_{\mathrm{shuffle}}}, node noise scale μnoise\mu_{\mathrm{noise}}, and layer sparsity μsparsity\mu_{\mathrm{sparsity}}. These are combined with shared dataset parameters controlling the number of observed features μzx\mu_{z_x}, number of classes μnc\mu_{n_c}, mean number of categories μmean_cat\mu_{\mathrm{mean\_cat}}, categorical feature ratio μpcat\mu_{p_{\mathrm{cat}}}, missing-value ratio μpmissing\mu_{p_{\mathrm{missing}}}, missingness mechanism μmissingness_type\mu_{\mathrm{missingness\_type}}, class-balance concentration μclass_balance\mu_{\mathrm{class\_balance}}, weight scaling μweight_scaling\mu_{\mathrm{weight\_scaling}}, and input distribution μdist\mu_{\mathrm{dist}}. Integer-valued parameters are sampled from truncated normal distributions over the ranges shown in Table 6 and then rounded to the nearest integer, while continuous parameters are sampled from truncated normals on their admissible intervals. Categorical parameters are sampled with an ϵ\epsilon-greedy scheme: with probability 1−ϵ1-\epsilon we select the reference value and with probability ϵ\epsilon we sample uniformly from the remaining choices. We also implemented Tree-based SCMs, as in related literature, but found that they did not make a significant contribution to our RAD-TFM pipeline, so we exclude them here for brevity [31].

Table 6: MLP-based SCM parameter space and sampling distributions. Numeric parameters are sampled from truncated normal distributions centered at the corresponding mean hyperparameter. Categorical parameters are sampled with an ϵ\epsilon-greedy rule with ϵ=0.3⋅σ\epsilon= 0.3 \cdot\sigma.

ParameterTypeRangeDistribution
Mean hidden size μh\mu_hinteger[1,256][1, 256]TruncNorm(μh,10⋅σ)(\mu_h, 10 \cdot\sigma)
Mean number of layers μl\mu_linteger[1,12][1, 12]TruncNorm(μl,σ)(\mu_l, \sigma)
Mean input width μxin\mu_{x_{\mathrm{in}}}integer[1,15][1, 15]TruncNorm(μxin,2⋅σ)(\mu_{x_{\mathrm{in}}}, 2 \cdot\sigma)
Activation function μa\mu_acategorical{relu, elu, tanh, identity}ϵ\epsilon-greedy
Mean number of features μzx\mu_{z_x}integer[2,200][2, 200]TruncNorm(μzx,szx⋅σ)(\mu_{z_x}, s_{z_x} \cdot\sigma)
Mean number of classes μnc\mu_{n_c}integer[2,10][2, 10]TruncNorm(μnc,σ)(\mu_{n_c}, \sigma)
Mean number of categories μmean_cat\mu_{\mathrm{mean\_cat}}integer[2,120][2, 120]TruncNorm(μmean_cat,2⋅σ)(\mu_{\mathrm{mean\_cat}}, 2 \cdot\sigma)
Categorical feature ratio μpcat\mu_{p_{\mathrm{cat}}}continuous[0,1][0, 1]TruncNorm(μpcat,0.1⋅σ)(\mu_{p_{\mathrm{cat}}}, 0.1 \cdot\sigma)
Shuffled categorical ratio μpshuffle\mu_{p_{\mathrm{shuffle}}}continuous[0,1][0, 1]TruncNorm(μpshuffle,0.1⋅σ)(\mu_{p_{\mathrm{shuffle}}}, 0.1 \cdot\sigma)
Missing-value ratio μpmissing\mu_{p_{\mathrm{missing}}}continuous[0,1][0, 1]TruncNorm(μpmissing,0.1⋅σ)(\mu_{p_{\mathrm{missing}}}, 0.1 \cdot\sigma)
Missingness type μmissingness_type\mu_{\mathrm{missingness\_type}}categorical{mcar, featurewise, self_masking}ϵ\epsilon-greedy
Class-balance concentration μclass_balance\mu_{\mathrm{class\_balance}}continuous[0.1,10][0.1, 10]TruncNorm(μclass_balance,1.5⋅σ)(\mu_{\mathrm{class\_balance}}, 1.5 \cdot\sigma)
Node noise scale μnoise\mu_{\mathrm{noise}}continuous[0,1][0, 1]TruncNorm(μnoise,0.1⋅σ)(\mu_{\mathrm{noise}}, 0.1 \cdot\sigma)
Layer sparsity μsparsity\mu_{\mathrm{sparsity}}continuous[0,0.9][0, 0.9]TruncNorm(μsparsity,0.1⋅σ)(\mu_{\mathrm{sparsity}}, 0.1 \cdot\sigma)
Weight scaling μweight_scaling\mu_{\mathrm{weight\_scaling}}categorical{None, Small, Large}ϵ\epsilon-greedy
Input distribution μdist\mu_{\mathrm{dist}}categorical{exponential, uniform, normal}ϵ\epsilon-greedy

Table 6.

Learned SCM Parameter Configurations

Different Domains Require Different SCM Parameters. In Table 7, we show a subset of the parameters used to configure the SCM generators, which determine the shape, geometry, and learnability of the generated datasets. For each benchmark, we show the parameters corresponding to the setting with the highest weight, as in Equation (5), the optimal response of the adversary when constrained to generate data that is similar to the real-world reference data. In the table, SCM size refers the dimensions of the MLP generating the data, which we split into five categorical values, balance refers to the class imbalance (higher corresponds to greater class balance), and noise refers to the scale of the noise added to the output of each neuron. We can clearly observe the variety in parameterizations across benchmarks, some of which align with what we know about the real-world datasets from that domain. For example, for the Genomics benchmark, most datasets have d=500d = 500 numeric features, and the domain-adaptation algorithm has correctly constrained the parameter space to datasets with many features (we limit the number of generated features to d=200d = 200 per dataset). While some parameters, such as the number of features, may be easier to interpret and adjust manually, for others, such as network sparsity or the activation function, it would be difficult to interpret or understand how changing these values would alter the generated datasets. We can see however, both in Table 7 and Figure 3, that our domain-adaptation algorithm is not only able to adjust these parameters automatically to align with and cover the real-world datasets in the meta-feature space, but that it is necessary to do so, as their values vary significantly between domains.

Table 7: SCM generator parameters with the highest weight in the first iteration of the RAD-TFM algorithm for various benchmarks. Different benchmarks correspond to significantly different data generation regimes. We report only a subset of the parameters in this table.

BenchmarkSCM Size# Feat.# ClassAct.Sparsitypcatp_{\mathrm{cat}}Cat. Dim.pmissp_{\mathrm{miss}}BalanceNoise
TabPertNet[40]Medium132tanh0.90.0-0.430.0
TabArena[14]Medium442tanh0.90.5500.230.0
PMLB[26]Small1210tanh0.90.0-0.230.0
EngineeringLarge652elu0.00.0-0.0100.01
GenomicsLarge2002relu0.10.0-0.430.05
HealthcareSmall52identity0.01.020.2100.5

Table 7.

Generator Coverage on Foreign-Exchange Data

Effect of the Generator Family. To investigate the limitations imposed by the synthetic generator family, we consider a corpus of 64 foreign-exchange datasets from OpenML [4], derived from daily candlestick price series. These datasets exhibit strong temporal dependencies that are not naturally represented by the non-temporal SCMs used in our main experiments. With the original generators, the entropy-regularized optimal transport alignment objective between the real and synthetic dataset embeddings is approximately Sd=2S_d = 2, compared with values below 0.2 on the other evaluated domains. We also observe no empirical coverage (C=0C = 0), where coverage is the fraction of reference datasets whose nearest synthetic embedding is at least as close as a resampled real-data neighbor, averaged over resampling trials.

To expand the generator family, we introduce a temporal SCM with controllable latent dimension, autoregressive persistence, feature-noise ratio, maximum lag, prediction horizon, and target signal strength. This extended parameter space reduces the alignment objective to approximately Sd=−0.6S_d = -0.6 and increases coverage to C=0.05C = 0.05. Although coverage remains limited, the additional temporal structure permits closer alignment with the reference domain. Table 8 reports the resulting mean test AUC. Neither fine-tuning nor RAD-TFM with the original generators improves upon the frozen backbone; in contrast, the temporal extension yields a modest improvement. The results illustrate that reweighting generators cannot compensate for missing data-generating mechanisms, while expanding the SCM family can extend the range of domains amenable to robust, domain-adapted continued training.

ModelMean AUCChange vs. base
TabPFN V20.6602—
TabPFN V2 (FT)0.6551−0.0051
TabPFN V2 (RAD-TFM; original SCMs)0.6597−0.0005
TabPFN V2 (RAD-TFM; temporal SCM)0.6652+0.0050

Table 8. Mean test AUC across 64 foreign-exchange datasets, comparing the original and expanded synthetic generator families.

Additional Experiment Details

In Tables 9–16 we provide the full experiment results from the TabPertNet [40], TabArena [14], PMLB [26], and domain-specific OpenML [4] tabular benchmarks. For all benchmarks, we evaluate on classification datasets only, and limit our evaluation to datasets with at most n=10kn = 10k samples and p=500p = 500 features. We use the training data from each dataset during the domain-adaptation process to approximate the reference distribution for the given domain. We report our test results on N=70N = 70 datasets from TabPertNet, N=91N = 91 datasets from PMLB, N=21N = 21 datasets from TabArena, N=31N = 31 dataset for Healthcare, N=11N = 11 datasets for Genomics, and N=20N = 20 datasets for Engineering. The original genomics datasets have O(10k)O(10k) features, so we first preprocess these datasets by randomly selecting a subset of d=500d = 500 features and down-sampling. For TabArena, we use the same folds as defined in the original work and corresponding leaderboard. For the OpenML datasets, we use the same train/test splits as provided from the database. For TabPertNet, we use the same train/test split used in the original paper. We report ROC-AUC for binary classification tasks and AUC one-vs-one (OVO) for multi-class tasks.

Compute Resources. All experiments were run on a single 80GB A100 GPU together with up to 140 CPU cores during the robust parameter-search stage. Constructing the domain-adapted reference distribution for a benchmark required approximately 10 minutes of GPU time, while each max-min iteration of the RAD-TFM algorithm required approximately 1 GPU-hour; we used 5 epochs for Genomics, 10 epochs for Engineering and Healthcare, and 30 epochs for TabPertNet, TabArena, and PMLB. The CPU-intensive parameter-search stage used roughly 140 cores for about 6 hours per full training run.

Licensing for existing assets. Our experiments use existing open-source benchmarks, datasets, models, and software packages. We cite the original sources for each benchmark and model in the main paper. TabArena is used under its publicly released benchmark resources; its code repository is licensed under Apache License 2.0, and the associated dataset curation repository is licensed under the BSD 3-Clause License. The TabArena authors further state that datasets whose licenses do not allow sharing or use in an academic benchmark are excluded from the benchmark. PMLB is publicly available under the MIT License. OpenML data and metadata are made available under OpenML’s terms of use; OpenML states that empirical data and metadata are available under a CC-BY license, while individual dataset metadata include dataset-specific license fields. For TabPertNet, we use the publicly released dataset split from the original benchmark; the TP-BERTa [39] repository distributing the TabPertNet/OpenTabs subset states that the authors obtained permission to distribute the used subset. We use TabPFN-V2 weights under the Apache 2.0 license and the open-source TabICL V2 classifier under the BSD 3-Clause License. Optuna is licensed under the MIT License. We use all assets only for academic research and benchmarking, and we do not redistribute modified versions of these assets.

DatasetTabPFN V2TabICL V2Conventional baselines
RAD-TFMBaseFTRAD-TFMBaseFTCatBoostRFXGBLRMLP
BankNoteAuthenticati1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9990.998
Bank_Personal_Loan_M0.5970.5780.5780.6180.6190.6040.6000.6010.5990.6060.590
Breast_Cancer0.8820.8760.8760.8920.8880.8860.8640.8910.8390.8850.869
Customer_Behaviour0.9580.9580.9580.7690.7820.7730.9440.9680.9720.9440.940
Employee Satisfaction0.6550.4330.4410.4740.4310.4740.5430.4730.4360.4390.598
NFL0.7250.7310.7310.7230.7290.7240.7200.6790.6710.6700.685
TravelInsurancePredi0.7080.6700.6680.7160.6880.7010.7080.5460.6310.7150.723
UniversalBank0.5970.5780.5780.6180.6190.6040.6000.5930.5990.6060.590
audit_data0.9970.9980.9981.0001.0001.0001.0001.0001.0000.9920.981
audit_risk0.9970.9980.9981.0001.0001.0001.0001.0001.0000.9920.981
b_depressed0.4680.4470.4450.5570.5210.5320.5780.5740.5130.4460.601
bt_dataset_t31.0000.9990.9990.9970.9990.9990.9980.9950.9950.9690.814
loan0.6730.6590.6590.6970.7020.6850.6690.6910.6970.6770.654
mechanical_analysis1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9490.940
new_model1.0001.0001.0001.0001.0001.0001.0001.0001.0001.0001.000
piracydataset0.5730.5720.5710.6070.5930.6060.5920.6060.6070.6060.486
0284_bank8FM0.9940.9940.9940.9940.9940.9940.9940.9910.9920.9930.994
0292_cpu_small0.9800.9800.9800.9820.9820.9830.9790.9750.9780.9590.972
0312_cpu_act0.9870.9880.9880.9890.9880.9880.9840.9830.9860.9700.982
0345_delta_ailerons0.9770.9750.9750.9780.9780.9780.9740.9750.9720.9670.971
0356_delta_elevators0.9500.9490.9490.9520.9530.9520.9520.9460.9440.9450.949
0400_analcatdata_sup1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9380.750
0419_pm100.6430.4380.4330.7620.6400.6590.5670.6770.5700.4040.562
0435_strikes1.0001.0001.0001.0001.0001.0000.9960.9901.0000.6160.555
0437_quake0.5210.4320.4310.5640.4900.5660.4470.5000.4820.5630.531
0445_arsenic-male-bl0.7440.6000.6000.7560.6670.7000.5330.5000.7780.4110.711
0446_arsenic-female-0.8580.8450.8440.8370.8390.8450.8070.7370.7860.8140.663
0472_analcatdata_mar0.6630.3600.3660.3770.6690.6060.5710.5260.4800.3830.429
0509_pollen0.5120.5000.5020.5300.5120.5230.5030.4980.5300.4560.528
0526_colleges_aaup1.0001.0001.0000.9780.9730.9741.0001.0000.9990.9670.961
0541_plasma_retinol0.5850.5910.5910.5850.6020.5910.6360.6990.3520.6250.511
0555_socomob0.9920.9870.9870.9480.9590.9550.9950.9890.9920.9270.931
0885_compas-two-year0.7090.7090.7090.7120.7050.7080.6930.4410.5700.7030.684
0948_Ishwar0.9960.9980.9980.9500.9460.9540.9900.9830.9880.9750.979
1011_cleve0.9430.9330.9330.9330.9380.9380.9290.9430.9330.9570.971
1142_Sick_numeric0.9950.9910.9910.9980.9970.9970.9950.9830.9780.9170.578

Table 9. TabPertNet [40] Test Results, Part 1 (AUC ↑\uparrow).

DatasetTabPFN V2TabICL V2Conventional baselines
RAD-TFMBaseFTRAD-TFMBaseFTCatBoostRFXGBLRMLP
1201_Gender-Recognit1.0001.0001.0000.9990.9990.9990.9980.9990.9990.9960.993
1222_premier_league_0.6950.6990.6990.6950.6900.6910.5770.6340.6180.6930.661
1408_national-longit1.0001.0001.0000.9960.9970.9961.0001.0001.0000.9920.990
1413_shill-bidding1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9960.997
1458_kdd_ipums_la_970.9380.9370.9370.9380.9390.9360.9390.9320.9230.8900.909
1461_heart-failure0.8610.8610.8610.8610.8780.8940.8220.8440.8560.9440.794
1512_eye_movements0.7140.7460.7450.8180.7470.8170.6900.6980.7070.5940.627
1564_Mammographic-Ma0.8170.8150.8150.8180.8180.8150.8080.7010.6580.8070.691
1578_kdd_ipums_la_970.9540.9530.9530.9560.9540.9550.9590.9540.9460.9070.927
1592_Diabetes-Data-S0.8730.8710.8710.8770.8730.8710.8760.8960.8710.8390.772
1600_VulNoneVul0.8010.7980.7970.7570.7620.7580.7200.6380.6980.7550.523
1635_Is-this-a-good-0.8100.8050.8040.7790.7870.7810.8200.7790.7520.7470.750
1692_Gender-Classifi0.9950.9950.9950.9930.9930.9940.9930.9920.9890.9930.993
1736_combined-wine-d1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9990.998
1742_Loan-Predicatio0.7850.7430.7430.7120.7330.7280.7660.7690.7370.7420.752
1752_Wisconsin-breas0.9960.9940.9940.9920.9920.9920.9900.9920.9870.9920.977
1759_Red_White-wine-1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9990.998
1898_Personal-Loan-M0.6180.6240.6230.6150.6190.6160.6310.6450.6030.6270.587
2304_electricity_see0.8750.8660.8660.8890.8870.8840.8530.8670.8750.6990.770
2305_electricity_see0.9340.9370.9370.9150.9130.9190.9050.9170.9220.7880.858
2306_electricity_see0.8770.8790.8790.8630.8580.8620.8620.8740.8710.7710.827
2308_electricity_see0.9120.9110.9110.9030.9020.9010.8910.8860.8860.7170.837
2389_airlines_seed_00.6790.6830.6830.6500.6440.6560.6400.6180.6370.6830.666
2390_airlines_seed_10.6460.6400.6400.6430.6380.6500.6370.6070.5850.6620.646
2391_airlines_seed_20.6500.6410.6410.6360.6290.6320.6190.6180.6500.6470.635
2392_airlines_seed_30.6070.6230.6230.5830.5850.5800.6070.6000.5640.6120.617
2393_airlines_seed_40.6770.6800.6790.6860.6860.6860.6470.6390.6070.6900.682
2619_sf-police-incid0.4650.4500.4500.5960.6040.6010.5300.5950.4100.4950.445
2620_sf-police-incid0.5090.4870.4850.4640.4560.4550.5180.5100.4570.5120.456
2621_sf-police-incid0.4780.5400.5440.6180.5700.5690.7310.7390.6160.6460.449
2622_sf-police-incid0.5220.4580.4580.6230.3860.3610.4820.4420.4300.4490.467
2703_compas-two-year0.7290.7300.7300.7330.7290.7330.7110.5250.5850.7320.733
trial1.0001.0001.0001.0001.0001.0001.0001.0001.0001.0001.000
wines_SPA1.0001.0001.0001.0001.0001.0001.0001.0001.0001.0001.000

Table 10. TabPertNet [40] Test Results, Part 2 (AUC ↑).

DatasetTabPFN V2TabICL V2Conventional baselines
RAD-TFMBaseFTRAD-TFMBaseFTCatBoostRFXGBLRMLP
Bank_Customer_Churn0.8760.8750.8740.8410.8250.8020.8780.8450.8450.7450.848
Fitness_Club0.8110.8100.8100.7790.7780.7770.8030.7600.7570.7960.563
Is-this-a-good-custo0.7310.7070.7120.7520.7490.7460.7290.6550.6700.7120.585
MIC0.6430.6250.6340.6740.6840.6780.6840.6780.6520.7220.635
Marketing_Campaign0.9080.8970.8940.8500.8300.7930.9070.8940.9090.8370.647
NATICUSdroid0.9860.9840.9850.9870.9870.9870.9850.9780.9840.9820.981
anneal0.9921.0001.0000.9991.0001.0000.9580.9990.9940.9530.961
blood-transfusion-se0.7410.7290.7320.7410.7400.7440.6970.6630.6490.7440.723
churn0.9440.9400.9420.9420.9440.9460.9380.9270.9330.8140.882
coil2000_insurance_p0.7520.7460.7490.7650.7580.7620.7210.6890.7020.7350.696
credit-g0.7800.7810.7820.7720.7720.7740.7740.7710.7720.7730.704
diabetes0.8410.8420.8440.8450.8470.8450.8300.8280.8080.8390.781
hazelnut-spread-cont0.9820.9820.9830.9920.9920.9920.9590.9500.9680.5840.931
maternal_health_risk0.9620.9520.9520.9640.9660.9680.9520.9550.9580.8040.793
polish_companies_ban0.9670.9630.9660.9800.9800.9820.9700.9450.9630.5320.814
qsar-biodeg0.9300.9250.9300.9330.9350.9340.9250.9150.9200.9210.919
seismic-bumps0.7730.7820.7780.7630.7540.7620.7580.7610.7400.7660.602
splice0.9980.9970.9980.9980.9970.9980.9960.9970.9950.9920.988
students_dropout_and0.8690.8640.8660.8620.8510.8600.8570.8560.8690.8640.832
taiwanese_bankruptcy0.9480.9480.9500.9460.9490.9480.9410.9410.9400.5920.832
website_phishing0.9730.9730.9730.9780.9780.9790.9760.9700.9720.9260.869

Table 11. TabArena [14] Test Results (OVO AUC ↑).

DatasetTabPFN V2 RAD-TFMTabPFN V2 BaseTabPFN V2 FTTabICL V2 RAD-TFMTabICL V2 BaseTabICL V2 FTCatBoostRFXGBLRMLP
1028_SWD0.8410.8430.8430.8610.8680.8640.8610.8610.8610.8600.827
1029_LEV0.9020.9010.9020.9040.9150.9090.9170.9110.9160.8850.849
1030_ERA0.7880.7870.7870.7850.7820.7820.7420.7370.7390.7880.724
294_satellite_image0.9950.9950.9950.9960.9960.9960.9890.9880.9910.9570.985
GAMETES_Epistasis_2_0.7390.7380.7370.7400.7320.6940.6840.6010.6620.5300.464
GAMETES_Epistasis_2_0.8760.8710.8720.8830.8760.8970.8510.7110.8070.4980.666
GAMETES_Epistasis_3_0.4800.4820.4820.7040.4480.6900.6190.5630.6170.5390.465
GAMETES_Heterogeneit0.7310.7270.7270.7370.6890.7170.7680.7250.7690.5330.637
GAMETES_Heterogeneit0.7950.7830.7850.7890.7770.8120.7450.6200.6680.5130.535
Hill_Valley_with_noi0.5320.5110.5100.4900.4650.5320.4860.5580.4670.4960.502
Hill_Valley_without_0.5420.4740.4740.5420.4950.4620.4770.5870.4580.4960.495
agaricus_lepiota1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9751.000
allbp0.9950.9950.9950.9970.9960.9960.9910.9940.9930.9320.659
allhypo0.9860.9840.9840.9910.9940.9930.9820.9830.9810.8400.707
allrep0.9040.8990.8990.9260.9100.9150.8980.9290.8900.8210.456
analcatdata_authorsh1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9990.999
analcatdata_dmft0.6210.6050.6140.6020.6200.6030.5330.5170.5480.5970.449
analcatdata_germangs0.8390.8350.8370.9410.8530.9570.6600.5950.6600.6980.573
analcatdata_lawsuit1.0001.0001.0001.0001.0001.0001.0001.0001.0001.0000.987
ann_thyroid0.9980.9990.9991.0001.0001.0001.0001.0001.0000.7890.922
balance_scale0.9850.9860.9861.0001.0001.0000.9150.7540.8710.9410.852
banana0.9670.9670.9670.9670.9670.9680.9660.9600.9580.5010.945
breast_cancer0.8000.7740.7740.8520.7830.7910.8000.7830.5390.8000.626
breast_cancer_wiscon1.0001.0001.0001.0001.0001.0001.0001.0000.9971.0000.929
breast_cancer_wiscon0.9850.9860.9860.9860.9860.9880.9920.9870.9840.9860.997
bupa0.6710.6670.6670.7460.6580.6790.6500.6250.4960.6880.667
calendarDOW0.8550.8480.8460.8380.8690.8530.8360.8610.8430.8010.779
car_evaluation0.9990.9990.9991.0001.0001.0001.0000.9991.0000.9210.862
cars1.0001.0001.0001.0001.0001.0000.9310.9320.9290.7810.628
chess0.9990.9990.9991.0001.0001.0001.0001.0000.9990.9960.998
churn0.9740.9730.9730.9740.9690.9610.9640.9600.9690.7690.580
clean11.0001.0001.0001.0001.0001.0001.0001.0001.0001.0000.918
clean21.0001.0001.0001.0001.0001.0001.0001.0001.0001.0001.000
coil20000.7690.7630.7630.7590.7510.7610.7420.6560.7410.7230.601
congressional_voting1.0001.0001.0001.0001.0001.0001.0001.0000.9810.9980.976
contraceptive_method0.7290.7300.7300.7150.7240.7120.6880.7010.7020.6700.648
credit_approval_aust0.9590.9580.9580.9650.9620.9640.9600.9660.9440.9510.918
credit_approval_germ0.8440.8460.8460.8270.8380.8210.8290.7980.7330.8240.735

Table 12. PMLB [26] Test Results, Part 1 (OVO AUC ↑\uparrow).

DatasetTabPFN V2 RAD-TFMTabPFN V2 BaseTabPFN V2 FTTabICL V2 RAD-TFMTabICL V2 BaseTabICL V2 FTCatBoostRFXGBLRMLP
dis0.9991.0001.0000.9990.9970.9980.9960.9950.9980.7320.771
dna0.9950.9950.9950.9970.9970.9970.9970.9960.9980.9900.991
ecoli0.9600.9600.9600.9600.9590.9600.9550.9560.9470.9410.872
haberman0.7250.6830.6830.6920.6580.6750.6080.5290.3750.7170.875
heart_disease_clevel0.9040.8940.8940.9040.9090.9180.9040.9160.9040.9040.880
heart_disease_hungar0.9110.9000.9000.9320.9630.9740.9470.9450.8950.9050.574
horse_colic_outcome0.7640.7490.7530.8650.8550.8610.8120.8280.7920.7310.776
horse_colic_surgery0.8210.8100.8100.7860.8060.8210.7900.8040.7220.8410.587
hypothyroid0.9670.9680.9680.9730.9800.9780.9790.9790.9770.8650.546
ionosphere1.0000.9960.9960.9961.0001.0001.0001.0001.0000.9300.816
irish1.0001.0001.0001.0001.0001.0001.0001.0001.0000.8690.902
kr_vs_kp1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9930.998
led240.9480.9480.9480.9510.9520.9510.9410.9420.9290.9480.946
led70.9570.9570.9570.9580.9600.9590.9490.9470.9520.9580.955
mfeat_factors1.0001.0001.0001.0001.0001.0000.9991.0001.0001.0001.000
mfeat_fourier0.9920.9930.9930.9960.9960.9970.9860.9860.9880.9810.977
mfeat_karhunen1.0001.0001.0001.0001.0001.0000.9990.9990.9991.0000.999
mfeat_morphological0.9680.9690.9690.9790.9690.9780.9620.9550.9580.9600.954
mfeat_pixel1.0001.0001.0001.0001.0001.0001.0001.0001.0001.0000.999
mfeat_zernike0.9900.9900.9900.9950.9940.9970.9740.9730.9660.9820.983
mofn_3_7_101.0001.0001.0001.0001.0001.0001.0001.0001.0001.0001.000
monk11.0001.0001.0001.0001.0001.0001.0001.0001.0000.3740.760
monk21.0001.0001.0001.0001.0001.0001.0000.9681.0000.5590.544
monk31.0001.0001.0001.0001.0001.0001.0000.9981.0000.8710.871
mushroom1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9771.000
optdigits1.0001.0001.0001.0001.0001.0000.9991.0000.9990.9990.999
page_blocks0.9630.9630.9630.9630.9610.9730.9640.9660.9620.9030.962
parity5+51.0001.0001.0001.0000.7331.0000.5310.6511.0000.6430.333
pendigits1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9961.000
penguins1.0001.0001.0001.0001.0001.0001.0001.0001.0001.0000.901
phoneme0.9510.9540.9540.9730.9680.9710.9380.9530.9460.8350.887
profb0.6980.6700.6710.7000.7100.5800.6520.6710.6800.6560.579
ring0.9960.9960.9960.9970.9970.9970.9950.9940.9950.8370.993
saheart0.7670.7530.7570.7550.7760.7650.8090.7390.7120.7720.658
satimage0.9950.9950.9950.9960.9960.9960.9890.9880.9910.9570.985
schizo0.5780.5670.5670.5340.4590.4690.4440.4210.4430.5830.533
segmentation1.0001.0001.0001.0001.0001.0001.0001.0001.0000.9950.994
spambase0.9880.9890.9890.9920.9930.9930.9880.9800.9900.9760.971
spect0.8810.8870.8870.9250.8870.9000.8620.8750.8810.9190.875
spectf0.9140.9190.9190.9240.9550.9440.9490.9440.8990.8480.828
splice0.9970.9970.9970.9990.9990.9990.9980.9950.9980.9440.956
threeOf91.0001.0001.0001.0001.0001.0001.0001.0001.0000.9610.969
tic_tac_toe0.9980.9990.9991.0000.9980.9990.9990.9951.0000.6020.778
titanic0.8050.8040.8040.7910.7870.7890.7830.7280.7420.7130.705
tokyo10.9800.9770.9770.9810.9820.9810.9870.9840.9840.8830.951
twonorm0.9980.9980.9980.9980.9980.9980.9980.9970.9970.9980.998
vehicle0.9780.9780.9770.9920.9840.9870.9410.9490.9380.9410.818
waveform_210.9720.9720.9720.9730.9730.9720.9640.9660.9660.9720.967
waveform_400.9720.9730.9730.9730.9740.9730.9600.9670.9680.9660.964
wine_quality_red0.8140.8220.8220.8520.8590.8560.8210.8710.8490.7770.673
wine_quality_white0.8600.8760.8750.8950.8810.8840.7830.7990.8090.6390.635
xd61.0001.0001.0001.0001.0001.0001.0001.0001.0000.8860.996
yeast0.8730.8760.8750.8740.8880.8820.8530.8700.8700.8270.814

Table 13. PMLB [26] Test Results, Part 2 (OVO AUC ↑\uparrow).

DatasetTabPFNn.e._{\mathrm{n.e.}} (RAD-TFM)TabPFNn.e._{\mathrm{n.e.}} (Base)TabPFNn.e._{\mathrm{n.e.}} (Finetuned)CatBoostRFXGBLRMLP
anneal_21.0001.0001.0001.0001.0001.0000.9700.997
cnae-9_14680.9990.9980.9980.9980.9970.9970.9980.993
glass_410.9750.9640.9640.9520.9390.9370.8190.497
kc1_10670.8680.8610.8610.8400.8490.8020.7890.820
kc2_10630.8680.8640.8640.8920.8680.8350.7120.831
kc3_10650.7800.7740.7740.7500.8480.6850.5540.702
mc1_10560.9120.9220.9210.9900.9220.9920.9170.884
mw1_10710.8420.8680.8680.7190.7110.5880.8330.640
pc1_10680.8820.8750.8760.8750.8050.8210.6920.533
pc2_10690.9900.9900.9900.9930.9820.9850.0350.250
pc3_10500.8470.8250.8250.7850.8090.7310.6400.719
pc4_10490.9460.9300.9300.9380.9340.9490.9180.880
seismic-bumps_15000.9930.9930.9930.9660.9660.9800.9930.748
semeion_15010.9980.9960.9960.9970.9960.9950.9950.991
sonar_400.9630.9630.9630.9810.9540.9720.9260.843
steel-plates-fault_11.0001.0001.0001.0001.0001.0000.6981.000
steel-plates-fault_40.9800.9800.9800.9740.9730.9710.7370.950
wall-robot-navigatio1.0001.0001.0001.0001.0001.0000.9080.986
wall-robot-navigatio1.0001.0001.0001.0001.0001.0000.9861.000
wall-robot-navigatio1.0001.0001.0001.0001.0001.0000.9870.999

Table 14. Engineering [4] Test Results (OVO AUC ↑\uparrow).

DatasetTabPFNn.e._{\mathrm{n.e.}} (RAD-TFM)TabPFNn.e._{\mathrm{n.e.}} (Base)TabPFNn.e._{\mathrm{n.e.}} (Finetuned)CatBoostRFXGBLRMLP
AP_Breast_Omentum_110.9140.9140.9140.8820.8820.8790.8660.864
AP_Colon_Omentum_1141.0001.0001.0000.9960.9961.0000.9701.000
AP_Colon_Ovary_11530.9840.9860.9860.9840.9740.9720.9480.950
AP_Colon_Uterus_11600.9700.9620.9620.9700.9290.9510.9670.967
AP_Omentum_Kidney_110.9900.9900.9900.9950.9900.9950.9710.971
AP_Omentum_Lung_11320.9900.9900.9900.9810.9900.9620.9620.971
AP_Omentum_Ovary_1150.9750.9560.9560.9750.9440.9440.8620.644
AP_Omentum_Uterus_111.0001.0001.0001.0001.0001.0000.9621.000
OVA_Colon_11610.9210.9180.9180.9410.9480.9320.9120.932
OVA_Omentum_11390.9910.9860.9860.9820.9600.9750.7110.944
madelon_14850.9490.9410.9420.9070.8450.8910.6070.625

Table 15. Genomics [4] Test Results (AUC ↑\uparrow).

DatasetTabPFNn.e._{\mathrm{n.e.}} (RAD-TFM)TabPFNn.e._{\mathrm{n.e.}} (Base)TabPFNn.e._{\mathrm{n.e.}} (Finetuned)CatBoostRFXGBLRMLP
SPECTF_16000.8440.8520.8520.8440.8250.8090.7800.728
SPECTF_3370.9490.9350.9360.9500.9510.9240.9280.717
SPECT_3360.8650.8620.8620.8540.8280.8310.8470.767
abalone_7200.9050.9050.9050.8920.8890.8800.8880.838
audiology_9990.9901.0001.0001.0001.0000.9711.0000.902
blood-transfusion-se0.7750.7710.7710.7620.7730.7290.7780.733
bodyfat_7781.0001.0001.0001.0001.0001.0000.9470.858
breast-cancer-droppe0.5380.5320.5320.4500.4210.4040.5030.456
breast-cancer_130.6960.6550.6550.6190.5950.6550.6130.589
cardiotocography_1461.0001.0001.0001.0001.0001.0000.9781.000
cardiotocography_1561.0001.0001.0001.0001.0001.0000.9751.000
chatfield_4_8200.9930.9860.9860.9790.9750.9070.9860.479
dermatology_10101.0001.0001.0001.0001.0001.0001.0001.000
haberman_430.8260.7990.8040.7340.6170.5650.8210.679
heart-c_9820.9240.9330.9330.9160.8910.8870.9200.882
heart-h_15650.6080.6060.6070.7050.6940.6590.5410.641
heart-h_9630.8560.8470.8470.8370.7920.7750.8130.804
mfeat-factors_9781.0001.0001.0001.0000.9970.9990.9990.997
mfeat-fourier_9711.0001.0001.0001.0001.0001.0001.0001.000
mfeat-karhunen_10201.0001.0001.0001.0001.0000.9990.9980.996
mfeat-morphological_1.0001.0001.0001.0001.0001.0001.0001.000
mfeat-pixel_10221.0001.0001.0001.0001.0001.0000.9960.999
mfeat-zernike_9951.0000.9990.9991.0000.9991.0000.9720.968
mu284_8801.0001.0001.0001.0001.0001.0001.0000.738
pbc_8100.7530.7570.7570.7780.7650.8310.7050.703
pcbse_8020.8330.8570.8550.9020.9010.9330.7280.666
pwLinear_7210.9500.9600.9600.9800.9801.0000.9700.700
thoracic-surgery_1500.7680.7110.7110.7460.7890.8790.7040.750
thoracic_surgery_4320.7680.7750.7710.7960.7300.7680.7680.571
vertebra-column_15230.9440.9330.9330.9470.9490.9000.9020.783
vertebra-column_15240.9620.9620.9620.9520.9310.9240.9670.824

Table 16. Healthcare [4] Test Results (AUC ↑\uparrow).

Additional Analyses

TabICL V2 Across Benchmarks. Table 17 reports the mean normalized AUC of the frozen, fine-tuned, and RAD-TFM-trained TabICL V2 classifier separately for each benchmark. In all three cases, the full method improves on both the original backbone and fine-tuning. These experiments use the same synthetic-generation and robust parameter-selection pipeline as TabPFN V2, without architecture-specific modifications. Normalization is performed over the expanded model comparison and therefore differs from the normalization used in Tables 1 and 2. Per-dataset AUC results for both TFM backbones and all five conventional baselines are reported in Tables 9–13.

BenchmarkTabICL BaseTabICL FTTabICL (RAD-TFM)
PMLB0.8920.9110.924
TabPertNet0.7480.7540.779
TabArena0.8910.8940.914

Table 17. Mean normalized AUC for TabICL V2 variants on the three principal benchmarks.

Training-Budget Sensitivity. To control for the number of training datasets, the fine-tuning runs use the same batch size and number of optimizer updates as the corresponding RAD-TFM runs. The models are evaluated at checkpoints spaced by 500 training datasets. We report the mean test-AUC gap between each trained model and its frozen backbone, evaluated on the respective reference benchmark. In particular, continued fine-tuning on the real-data distribution degrades sharply beyond the earliest checkpoints, whereas the synthetic distribution selected by RAD-TFM maintains positive gains throughout training (Table 18). These comparisons control for dataset exposures and parameter-update counts, rather than total wall-clock time. The 30k RAD-TFM entries correspond to selected checkpoints from training runs that processed 30k generated datasets in total, rather than evaluations at the terminal checkpoint. The fine-tuning entries at 30k report performance evaluated at that checkpoint.

BenchmarkMethod5001k1.5k2k6k10k30k
TabArenaRAD-TFM+0.0005+0.0010+0.0016+0.0015+0.0026+0.0028+0.0040*
TabArenaFine-tuning+0.0011+0.0015-0.0003-0.0024-0.0505-0.0840-0.0844
PMLBRAD-TFM+0.0019+0.0025+0.0026+0.0033+0.0029+0.0030+0.0030*
PMLBFine-tuning-0.0006-0.0012-0.0017-0.0023-0.0166-0.0199-0.0233
TabPertNetRAD-TFM+0.0031+0.0049+0.0052+0.0058+0.0083+0.0105+0.0166*
TabPertNetFine-tuning+0.0019+0.0009+0.0010+0.0001-0.0020-0.0097-0.0103

Table 18. Mean raw test-AUC change relative to frozen TabPFN V2 at different training-dataset budgets. Positive values indicate improvement. At 30k, RAD-TFM entries marked with an asterisk correspond to selected checkpoints from runs that processed 30k generated datasets, whereas fine-tuning entries are evaluated at the 30k checkpoint.

Individual and Combined Contributions. To further quantify the role of the two components, we measure the raw-AUC gain over the frozen model from domain adaptation alone (ΔDA\Delta_{\mathrm{DA}}), robustness alone (ΔR\Delta_{\mathrm{R}}), and their combination (ΔRAD\Delta_{\mathrm{RAD}}). The residual ΔRAD−ΔDA−ΔR\Delta_{\mathrm{RAD}}-\Delta_{\mathrm{DA}}-\Delta_{\mathrm{R}} captures the deviation from additivity, as shown in Table 19. The average gain under the full model (+0.0055) is close to the sum of the individual average gains (+0.0052); the corresponding residual is +0.0003. While the interaction varies across benchmarks, the two components contribute complementary improvements on average. Paired one-sided Wilcoxon signed-rank tests of normalized AUC over 236 datasets give p=10−4p=10^{-4} for domain adaptation alone, p=0.028p=0.028 for robustness alone, and p<10−4p<10^{-4} for the combined method, each relative to the frozen model.

BenchmarkΔDA\Delta_{\mathrm{DA}}ΔR\Delta_{\mathrm{R}}ΔDA+ΔR\Delta_{\mathrm{DA}} + \Delta_{\mathrm{R}}ΔRAD\Delta_{\mathrm{RAD}}Residual
PMLB+0.0033+0.0021+0.0053+0.0030-0.0023
TabPertNet+0.0074+0.0094+0.0167+0.0166-0.0001
TabArena+0.0013+0.0018+0.0031+0.0040+0.0010
Genomics+0.0024+0.0004+0.0028+0.0037+0.0009
Healthcare+0.0021+0.0023+0.0044+0.0035-0.0008
Engineering+0.0001-0.0016-0.0015+0.0020+0.0035
Mean+0.0028+0.0024+0.0052+0.0055+0.0003

Table 19. Change in mean raw AUC from the frozen TabPFN V2 and the residual from additive contributions.

NeurIPS Paper Checklist

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction state the paper’s theoretical, algorithmic, and empirical contributions, and these are matched by the formal development in Sections 2–4 and the experiments in Section 5.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses several concrete limitations, including limited statistical power for small benchmarks, challenging domains where the current SCM space and metafeatures may be insufficient, and design restrictions such as fixed generated sample sizes; see Section 5 and Appendices § 8 and § 11.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theoretical formulation and assumptions are introduced in Section 2, and formal proofs and supporting derivations are provided in Appendix § 10.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper specifies the algorithmic pipeline, benchmark construction, evaluation protocol, hyperparameters, and detailed pseudocode in Section 5 and Appendices § 8, § 9, § 11, and § 14.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not release code or additional assets with the current submission. The experiments rely on publicly available benchmarks and pretrained model backbones, and the method, optimization objective, parameter space, training protocol, and evaluation procedure are described in the paper and its appendices.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: Section 5 gives the main training setup and benchmark choices, and Appendices B, D, and G provide additional details on splits, hyperparameters, and implementation choices.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Rather than error bars, we report formal statistical significance tests for the main claims, including Friedman tests and pairwise Wilcoxon signed-rank tests, as described in Section 5 and detailed further in Appendix G. The tests span six benchmarks, with additional paired comparisons and bootstrap confidence intervals reported for the second TFM backbone and the component ablations.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper specifies the compute configuration and runtime breakdown used in practice, including GPU type and memory, CPU usage, domain-adaptation cost, and benchmark-dependent training duration; see Appendix G.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: To the best of our knowledge, this work conforms to the NeurIPS Code of Ethics. The paper studies generic tabular classification methods using cited benchmark resources and does not involve human subjects or the release of sensitive assets.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The submission does not contain a dedicated broader-impact discussion. The work is methodological and application-agnostic, focusing on a generic tabular classifier rather than a specific downstream deployment setting.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: The paper does not release high-risk generative models, scraped datasets, or similar assets that would require a safeguards discussion for controlled release.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] Justification: We use existing benchmarks, datasets, software packages, and pretrained model weights. We cite the original creators in the main paper and provide an appendix paragraph listing the relevant licenses and terms of use for the principal assets, including TabArena, PMLB, OpenML, TabPertNet, Optuna, TabPFN-V2, and TabICL V2 in Appendix G. We use these assets only for academic research and benchmarking and do not redistribute modified versions of the assets.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: We do not release new code, models, datasets, or other assets as part of the current submission.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: This work does not involve crowdsourcing or research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: This work does not involve human subjects, so IRB approval and related participant-risk disclosures are not applicable.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: LLMs are not an important, original, or non-standard component of the core methodology studied in this paper.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework, July 2019. arXiv:1907.10902 [cs].arxiv.org/abs/1907.10902
  2. [2]Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should we really use post-hoc tests based on mean-ranks? The Journal of Machine Learning Research, 17(1):152–161, 2016.DOI
  3. [3]James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
  4. [4]Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn, and Joaquin Vanschoren. OpenML Benchmarking Suites, November 2021. arXiv:1708.03731 [stat].
  5. [5]Pavel Brazdil, Christophe Giraud-Carrier, Carlos Soares, and Ricardo Vilalta. Metalearning: Applications to data mining. Springer, 2009.DOI
  6. [6]Leo Breiman. Random Forests. Machine Learning, 45(1):5–32, October 2001.
  7. [7]Kimberly Villalobos Carballo, Liangyuan Na, Yu Ma, Léonard Boussioux, Cynthia Zeng, Luis R. Soenksen, and Dimitris Bertsimas. TabText: A Flexible and Contextual Approach to Tabular Data Representation, July 2023. arXiv:2206.10381 [cs].
  8. [8]Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, New York, NY, USA, August 2016. Association for Computing Machinery.DOI
  9. [9]Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. Generating multi-label discrete patient records using generative adversarial networks, 2018.arxiv.org/abs/1703.06490
  10. [10]William Jay Conover. Practical nonparametric statistics. john wiley & sons, 1999.
  11. [11]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  12. [12]Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1–30, 2006.DOI
  13. [13]Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. CatBoost: gradient boosting with categorical features support, October 2018. arXiv:1810.11363 [cs].arxiv.org/abs/1810.11363
  14. [14]Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A Living Benchmark for Machine Learning on Tabular Data, June 2025. arXiv:2506.16791 [cs].arxiv.org/abs/2506.16791
  15. [15]Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811, 2018.arxiv.org/abs/1807.02811
  16. [16]Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189–1232, October 2001. Publisher: Institute of Mathematical Statistics.DOI
  17. [17]Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Müller, and Frank Hutter. Real-tabpfn: Improving tabular foundation models via continued pre-training with real-world data, 2025.arxiv.org/abs/2507.03971
  18. [18]Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting Deep Learning Models for Tabular Data, October 2023. arXiv:2106.11959 [cs].arxiv.org/abs/2106.11959
  19. [19]Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on tabular data?, July 2022. arXiv:2207.08815 [cs].arxiv.org/abs/2207.08815
  20. [20]Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. TabLLM: Few-shot Classification of Tabular Data with Large Language Models. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, pages 5549–5581. PMLR, April 2023. ISSN: 2640-3498.arxiv.org/abs/2210.10723
  21. [21]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025.DOI
  22. [22]Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second, September 2023. arXiv:2207.01848 [cs].arxiv.org/abs/2207.01848
  23. [23]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, January 2025. Publisher: Nature Publishing Group.DOI
  24. [24]Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks, September 2019. arXiv:1706.06083 [stat].arxiv.org/abs/1706.06083
  25. [25]Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Benjamin Feuer, Chinmay Hegde, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When Do Neural Nets Outperform Boosted Trees on Tabular Data?, July 2024. arXiv:2305.02997 [cs].
  26. [26]Randal S Olson, William La Cava, Patryk Orzechowski, Ryan J Urbanowicz, and Jason H Moore. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData mining, 10(1):36, 2017.arxiv.org/abs/1703.00512
  27. [27]Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment, 11(10):1071–1083, 2018.DOI
  28. [28]Judea Pearl. Causality. Cambridge University Press, Cambridge, 2 edition, 2009.
  29. [29]Fábio Pinto, Carlos Soares, and Joao Mendes-Moreira. Towards automatic generation of metafeatures. In Pacific-Asia conference on knowledge discovery and data mining, pages 215–226. Springer, 2016.
  30. [30]Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 102477–102555. PMLR, 2026.arxiv.org/abs/2602.11139
  31. [31]Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data, May 2025. arXiv:2502.05564 [cs].arxiv.org/abs/2502.05564
  32. [32]Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659, 2019.DOI
  33. [33]Adriano Rivolli, Luís P. F. Garcia, Carlos Soares, Joaquin Vanschoren, and André C. P. L. F. de Carvalho. Characterizing classification datasets: a study of meta-features for meta-learning, 2019.arxiv.org/abs/1808.10406
  34. [34]Ramalingam Shanmugam. Elements of causal inference: foundations and learning algorithms. Journal of Statistical Computation and Simulation, 88(16):3248–3248, November 2018.DOI
  35. [35]Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training, June 2021. arXiv:2106.01342 [cs].arxiv.org/abs/2106.01342
  36. [36]Shuhei Watanabe. Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance, May 2023. arXiv:2304.11127 [cs].arxiv.org/abs/2304.11127
  37. [37]Yulun Wu and Doron L. Bergman. Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer, June 2025. arXiv:2502.04573 [cs].arxiv.org/abs/2502.04573
  38. [38]Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan, 2019.arxiv.org/abs/1907.00503
  39. [39]Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Z. Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making Pre-trained Language Models Great on Tabular Prediction, March 2024. arXiv:2403.01841 [cs].arxiv.org/abs/2403.01841
  40. [40]Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Towards Cross-Table Masked Pretraining for Web Data Mining, February 2024. arXiv:2307.04308 [cs].arxiv.org/abs/2307.04308
  41. [41]Han-Jia Ye, Si-Yang Liu, and Wei-Lun Chao. A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities, June 2025. arXiv:2502.17361 [cs].arxiv.org/abs/2502.17361
  42. [42]Xiyuan Zhang and Danielle Maddix Robinson. Mitra: Mixed synthetic priors for enhancing tabular foundation models, July 2025.

Paper details

vlnc:W_a5jztzpk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents