betaFeedback

WorksComputer Science

FTFM: Factorized Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) are becoming a powerful alternative to conventional tabular learning, adapting to new datasets in context without task-specific training. Current architectures, however, face a trade-off: models that preserve cell-level representations are expensive because they repeatedly attend across rows and columns, while faster models collapse each row into a fixed embedding and lose this fine-grained state. We introduce Factorized Tabular Foundation Models (FTFM), which preserve cell representations while factorizing their computation through one contextual latent per row and one per column. Cells no longer attend to one another directly; instead, each cell is updated through the interaction of its row and column latents, reducing repeated attention while retaining feature-specific information. FTFM achieves competitive predictive performance in tabular classification on major benchmarks such as TabArena and BeyondArena.

https://github.com/machinelearningnuremberg/ftfm-official

Introduction

A tabular foundation model is a single network that reads a labeled training set together with the test rows and returns predictions in one forward pass, without a gradient step on the target dataset. Its parameters are meta-learned once, over datasets drawn from a prior, and are then held fixed for every table it is applied to (§ 3.1). Since TabPFN (Hollmann et al., [14] showed that such a model outperforms tuned gradient-boosted ensembles, models of this type have occupied the leading positions of the public benchmarks ([10, 28, 5]).

Two architectural designs are in use by the state-of-the-art models, and both of them are variations of the classical Transformer architecture for tabular data. Feature-space preserving models ([15, 5]) maintain a grid of cell representations and update it with attention that alternates between rows and columns. These models force the row attention to be executed once per column and the column attention once per row. Therefore, for a table with NN rows and MM features at width DD, one layer costs Θ(MN2D+NM2D+NMD2)\Theta(MN^2D + NM^2D + NMD^2), which is prohibitive on large tables. Feature-space collapsing models ([27, 28, 10] (TabFM Team 2026)) compress each row into a fixed-dimensional embedding before the in-context stage. The compression layers, however, retain both row and column cell-grid attentions as pre-ICL feature transformers. The attention over rows is computed through a small set of learned inducing points and so is linear in NN, but this is an approximation of the attention it replaces rather than a cheaper exact computation. In contrast, the attention over the MM features of a row is not reduced at all and still costs Θ(NM2D)\Theta(NM^2D). Such a collapse is also a projection of the feature space onto a static representation shared across all datasets, after which no cell-level state remains, so any information the in-context stage requires must pass a pre-ICL bottleneck whose width was fixed at pretraining time. This practice therefore trades speed against representational loss.

We fix tabular models with a novel design, which we call factorized tabular models (FTFM). At every layer, the table is summarized by one latent vector per row and one per column, and each cell representation is updated as a factorization of the latents of its own row and its own column through a learned interaction (§ 3.2). The construction generalizes classical matrix factorization ([19]) and replaces cell-to-cell attention with row-to-row and column-to-column interaction. The new paradigm drops the per-layer cost to Θ(N2D+M2D+NMD2)\Theta(N^2D + M^2D + NMD^2), comparable to the featurecollapsing architectures. Therefore, our architecture offers (i) the expressivity of feature-preserving models, and (ii) the reduced inference runtime of the feature-collapsing architectures.

Furthermore, our modeling comes at no sacrifice in empirical predictive performance. We provide extensive experiments in terms of classification performance on the two most popular benchmarks for table prediction, namely TabArena (Erickson et al., [6]) (38 datasets) and BeyondArena (Purucker et al., [26]) (98 datasets). FTFM achieves competitive performance on both benchmarks. Our model is a novel architecture for tabular data that is able to match state-of-the-art results.

We contribute (i) FTFM, a novel family of architectures based on deep table factorizations; (ii) a mechanism that preserves the feature space at the inference complexity of architectures that collapse features; and (iii) strong experiments on TabArena and BeyondArena rivaling the state-of-the-art foundation models.

Related work

Tabular foundation models. Meta-learning one network across many tables predates the current wave of models. The first tabular foundation model is Dataset2Vec (Jomaa et al., [18]) (arXiv 2019; published 2021), a Deep-Set architecture meta-learned over tables to produce latent representations of whole datasets. In today’s vocabulary, it is a self-supervised world model of tables and not a supervised predictor, so it was never compared against a classifier. Iwata & Kumagai [17] introduced the first tabular foundation model for supervised prediction, again built on Deep Sets. It did not displace gradient-boosted trees ([3, 25]), which at the time still beat deep models on mid-sized benchmarks (Grinsztajn et al., [8]). TabPFN (Hollmann et al., [14]) was the first meta-learned supervised model to outperform tuned tree ensembles on small classification benchmarks, using a Transformer meta-learned over millions of synthetic datasets drawn from a prior (Müller et al., [24]). Its successors held the state of the art for several years, most recently TabPFN-3 (Grinsztajn et al., [10]), which has since been rivaled by TabFM (TabFM Team, 2026) and EXAONE Tabular (Eo et al., [5]). Among fully open models, where the architecture, the meta-learning code and the synthetic pretraining data are all released, TabICL (Qu et al., [27]) and its successor TabICLv2 (Qu et al., [28]) are the strongest.

The strongest models of this wave differ mainly in their treatment of the feature axis and fall into two families. Feature-space preserving models keep an N×MN \times M grid of cell representations at every layer and update it by attention that alternates between rows and columns, that is, axial attention (Ho et al., [13]) over the table. The pattern originates with Lorch et al. [22], who introduced alternating attention over the observation and variable axes of a data matrix for amortized causal structure learning, and was popularized for prediction by TabPFN-2 (Hollmann et al., [15]), followed by TabPFN-2.5 (Grinsztajn et al., [9]), EXAONE Tabular (Eo et al., [5]) and LimiX (Zhang et al., [36]). Feature-space collapsing models instead project each row onto a fixed number of shared feature dimensions and run in-context learning directly across rows. TabICL established the two-stage format of a feature representation stage followed by an ICL stage, and most current state-of-the-art models adopt it, among them TabPFN-3 and TabFM (TabFM Team, 2026). On the other hand, TabDPT (Ma et al., [23]) is a single-stage variant that embeds whole rows as tokens and adds retrieval. There are also exist LLM-based models, which serialize tables into text or build on pretrained language backbones ([12, 33, 7]). They exploit column-name semantics that the other two families discard, but have so far not matched native tabular architectures on standard benchmarks (Ma et al., [23]).

Matrix factorization. Latent-factor models of tables are older than tabular deep learning itself. In collaborative filtering, a rating matrix is approximated by inner products of row and column factors plus bias terms (Koren et al., [19]). Under a low-rank assumption, the missing entries are recoverable exactly (Candès & Recht, [2]). Factorization machines (Rendle, [29]) extend the construction to arbitrary feature vectors by assigning every feature a latent vector and scoring pairwise interactions with inner products. In all of these models, the factors are free parameters, estimated once per dataset. We retain the functional form of a cell explained by a row latent, a column latent and their interaction, but compute the latents by attention over the table itself. However, FTFM does not train a factorization of each table, but constructs it implicitly through in-context learning in a single pass.

Table factorizations. Neural networks that factorize tables have been explored by prior work. For instance, TabularNet [4] pools row-level and column-level states, next to a graph encoder over cell relations. Similarly, neural factorization machines [11] model products of fitted feature embeddings through an MLP, and DCN V2 [31] learns explicit low-rank feature crosses in a similar setting. However, these methods train the factorized matrices separately for each given dataset. In contrast, FTFM is a foundation model that introduces a novel architecture for computing the factorization of a table in a single forward pass, through in-context learning.

Efficient attention and scalable serving. The cost of attention over tables has been reduced in several directions. Inducing points [20] bottleneck an attention layer through a small learned set, and TabICLv2, TabFM, and TabPFN-3 apply them in their column stage. LimiX-2M [32] considers the opposite direction and reports a low-rank collapse of the cell representations in trained tabular models. Serving-oriented methods change the computed function to make in-context inference affordable, by compressing the context [34], by replacing softmax attention with a linear variant [35], or by dropping cell-level state in favor of long-context row attention [16]. Our factorization requires no such approximation on the cell grid, and the support-side computation is reusable across queries exactly.

Proposed method

Matrix factorization decomposes a matrix X∈RN×MX \in\mathbb{R}^{N \times M} with NN rows and MM columns via the product of a row-latent matrix U∈RN×DU \in\mathbb{R}^{N \times D}, a column-latent matrix V∈RM×DV \in\mathbb{R}^{M \times D}, row biases U0∈RNU^{0} \in\mathbb{R}^{N}, and column biases V0∈RMV^{0} \in\mathbb{R}^{M} [19]. Each cell is approximated as Xij≈⟨Ui,Vj⟩+Ui0+Vj0X_{ij} \approx\langle U_i,V_j\rangle+U_i^{0}+V_j^{0}. This section develops that principle into an architecture for tabular in-context learning. Tables are represented by row latents and column latents throughout the network, and every cell representation is a factorization of the two.

Tabular in-context learning

A supervised tabular task is a tuple D=(Xtr,ytr,Xte,yte)\mathcal{D} = (X^{\mathrm{tr}}, y^{\mathrm{tr}}, X^{\mathrm{te}}, y^{\mathrm{te}}) with training inputs Xtr∈RNtr×MX^{\mathrm{tr}} \in\mathbb{R}^{N^{\mathrm{tr}} \times M}, labels ytr∈YNtry^{\mathrm{tr}} \in\mathcal{Y}^{N^{\mathrm{tr}}}, and test inputs Xte∈RNte×MX^{\mathrm{te}} \in\mathbb{R}^{N^{\mathrm{te}} \times M}, where Y={1,…,K}\mathcal{Y} = \{1,\ldots,K\} for classification and Y=R\mathcal{Y} = \mathbb{R} for regression. A tabular foundation model is a single network fθf_{\theta} that maps the labeled training set and the test inputs directly to test predictions, y^te=fθ(Xtr,ytr,Xte)\hat{y}^{\mathrm{te}} = f_{\theta}(X^{\mathrm{tr}},y^{\mathrm{tr}},X^{\mathrm{te}}), with no parameter update on the target dataset. Its weights are meta-learned over a prior p(D)p(\mathcal{D}) over datasets, by minimizing ED∼p(D)[L(y^te,yte)]\mathbb{E}_{\mathcal{D}\sim p(\mathcal{D})}[\mathcal{L}(\hat{y}^{\mathrm{te}},y^{\mathrm{te}})] over θ\theta. Here L\mathcal{L} compares the predictions for all test rows against ytey^{\mathrm{te}}, using the mean cross-entropy for classification and a mean-squared error loss for regression, and p(D)p(\mathcal{D}) is in practice a structural-causal-model prior from which datasets are sampled on the fly [24, 27]. All knowledge about how to learn is contained in θ\theta.

Factorized table representations

Internally, fθf_{\theta} operates on latent representations of every cell. After stacking the N=Ntr+NteN = N^{\mathrm{tr}} + N^{\mathrm{te}} rows of a task into one table X∈RN×MX \in\mathbb{R}^{N \times M}, the network lifts it to a tensor Z(0)∈RN×M×DZ^{(0)} \in\mathbb{R}^{N \times M \times D} and transforms it further through layers Z(ℓ)Z^{(\ell)}, so that the representation of cell (i,j)(i,j) can capture the interactions of row ii with column jj. The architecture families of Section 2 differ in how Z(ℓ+1)Z^{(\ell+1)} is computed from Z(ℓ)Z^{(\ell)}. Preserving models let cells attend to cells along both axes, and collapsing models delete the cell axis after an initial pre-ICL projection.

Our construction proposes a different principle. At every layer ℓ\ell, the table is summarized by a row-latent matrix U(ℓ)∈RN×DU^{(\ell)} \in\mathbb{R}^{N \times D} and a column-latent matrix V(ℓ)∈RM×DV^{(\ell)} \in\mathbb{R}^{M \times D}, and each cell is updated by a factorization of its row and column latents,

Zij(ℓ+1)=Zij(ℓ)+g(ℓ)(Ui(ℓ),Vj(ℓ)),g(u,v)=MLP⁡([u;v;u⊙v]).(1)Z_{ij}^{(\ell+1)} = Z_{ij}^{(\ell)} + g^{(\ell)}(U_i^{(\ell)}, V_j^{(\ell)}), \qquad g(u,v) = \operatorname{MLP}([u \mathbin{\mathbin{;}} v; u \mathbin{\odot} v]). \tag*{(1)}

where [;;;][\mathbin{;}\mathbin{;}\mathbin{;}] denotes concatenation. Classical matrix factorization is the degenerate case of (1). Summing the Hadamard term recovers ⟨Ui,Vj⟩\langle U_i,V_j\rangle, so a single layer with g(u,v)=⟨u,v⟩g(u,v) = \langle u,v\rangle and the bias terms reproduces the decomposition above. Three aspects differ from the classical case. The reconstruction has DD dimensions per cell instead of one scalar. The interaction is learned and not fixed to an inner product. And the factors are not free parameters fitted to one dataset, but are computed by the network itself as follows:

Ui(ℓ)=SA⁡row(Zi,:(ℓ)),(2)U_i^{(\ell)} = \operatorname{SA}_{\mathrm{row}}\left(Z_{i,:}^{(\ell)}\right), \tag*{(2)}

pool row ii’s MM cells with a learned query,

Vj(ℓ)=SA⁡col(Z1:Ntr,j(ℓ)),(3)V_j^{(\ell)} = \operatorname{SA}_{\mathrm{col}}\left(Z_{1:N^{\mathrm{tr}},j}^{(\ell)}\right), \tag*{(3)}

pool column jj’s NtrN^{\mathrm{tr}} cells with a learned query,

U(ℓ)←U(ℓ)+MHA⁡(U(ℓ);U1:Ntr(ℓ)),(4)U^{(\ell)} \leftarrow U^{(\ell)} + \operatorname{MHA}\left(U^{(\ell)}; U_{1:N^{\mathrm{tr}}}^{(\ell)}\right), \tag*{(4)}

rows attend to training rows (ICL),

V(ℓ)←V(ℓ)+MHA⁡(V(ℓ);V(ℓ)),(5)V^{(\ell)} \leftarrow V^{(\ell)} + \operatorname{MHA}\left(V^{(\ell)}; V^{(\ell)}\right), \tag*{(5)}

columns attend to columns,

where SA denotes a pooling attention that summarizes a set of vectors through a single learned query, and MHA(AA; BB) denotes multi-head attention with queries from AA and keys and values from BB. The label yitry_i^{\mathrm{tr}} enters additively into the cells of training row ii, so the row latents of Equation (2) are label-aware and the attention of Equation (4) carries the in-context learning. The latent of a test row therefore retrieves information from the latents of the labeled rows.

In FTFM, no cell ever attends directly to another cell. All row-row interaction is carried by NN row latents and all column-column interaction by MM column latents, and the two are combined only inside gg. The cell grid and the feature space are preserved (i.e., Zℓ∈RN×M×DZ^\ell\in\mathbb{R}^{N \times M \times D}), but the computation is factorized.

The prediction of our model is a function of the row latents and the training labels only, inspired by Arbel et al. [1]. Splitting the final latents U(L)U^{(L)} into their training and test rows, UtrU^{\mathrm{tr}} and UteU^{\mathrm{te}},

y^te=1H∑h=1Hsoftmax⁡((UteWq(h))(UtrWk(h))Tdh)ytr,(6)\widehat{y}^{\mathrm{te}} = \frac{1}{H}\sum_{h=1}^{H}\operatorname{softmax}\left(\frac{\left(U^{\mathrm{te}}W_q^{(h)}\right)\left(U^{\mathrm{tr}}W_k^{(h)}\right)^{\mathsf{T}}}{\sqrt{d_h}}\right)y^{\mathrm{tr}}, \tag*{(6)}

where Wq(h),Wk(h)∈RD×dhW_q^{(h)}, W_k^{(h)} \in\mathbb{R}^{D \times d_h} and the softmax is taken over training rows; the prediction is computed as a head-averaged, attention-weighted mean of the training labels. The labels enter as values of the attention, one-hot encoded for classification and as the target value for regression, so Equation (6) is the same operation for both task types, and the network fθf_\theta of § 3.1 is the composition of the cell embedder, the LL factorized blocks, and Equation (6).

Asymptotic advantage of the factorization

The tabular foundation model families differ in the interaction terms, that is, in the attention scores. Feature-space preserving models let the NN cells of a column attend to one another, once per column, Θ(MN2D)\Theta(MN^2D), and the MM cells of a row attend to one another, once per row, Θ(NM2D)\Theta(NM^2D). In addition, a feed-forward layer on each cell adds a Θ(NMD2)\Theta(NMD^2) complexity. A forward pass therefore costs Θ(MN2D+NM2D+NMD2)\Theta(MN^2D + NM^2D + NMD^2) ([15, 5]).

FamilyExamplesInference complexityFeature preservation
PreservingTabPFN-2, EXAONE TabularΘ(MN2D+NM2D+NMD2)\Theta(MN^2D + NM^2D + NMD^2)yes
CollapsingTabICLv1/v2, TabPFN-3, TabFMΘ(N2D+NM2D+NMD2)\Theta(N^2D + NM^2D + NMD^2)no
Factorized (ours)Θ(N2D+M2D+NMD2)\Theta(N^2D + M^2D + NMD^2)yes

Table 1. Asymptotic cost of one in-context forward pass

Feature-space collapsing models retain both of these attentions as pre-ICL feature transformers and add an ICL stage over the collapsed row embeddings, Θ(N2D)\Theta(N^2D). In the pre-ICL stage, the attention over the NN cells of a column is routed through II learned inducing points, so those cells never attend to one another directly and that attention costs Θ(NMID)\Theta(NMID), which is Θ(NMD)\Theta(NMD) for constant II. The attention over the MM features of a row is not reduced and remains Θ(NM2D)\Theta(NM^2D). The feed-forward layer at the cell level adds a complexity of Θ(NMD2)\Theta(NMD^2). One forward pass costs Θ(N2D+NM2D+NMD2)\Theta(N^2D + NM^2D + NMD^2) ([27, 28]).

In FTFM, the row latents attend to the training rows, NN of them in total rather than once per column, Θ(N2D)\Theta(N^2D), and the column latents attend to the column latents, MM in total rather than once per row, Θ(M2D)\Theta(M^{2}D). We also apply cell feed-forward layers for a complexity of Θ(NMD2)\Theta(NMD^{2}). One forward pass costs Θ(N2D+M2D+NMD2)\Theta(N^{2}D + M^{2}D + NMD^{2}). FTFM has an asymptotic edge over other feature-preserving architectures, and is comparable to feature-collapsing models. Therefore, it maintains the feature space expressivity at the same asymptotic cost as feature-collapsing architectures.

Experiments

Experimental protocol

Model and pretraining. Our FTFM model has 121.6M parameters and consists of 16 factorized blocks of width D=512D = 512 with H=8H = 8 heads, feature groups of 3, robust per-column normalization, and the equivariant vote head. We meta-learn FTFM following a three-stage curriculum, each stage warm-started from the one before it (Table 2). Stage A adopts TabICLv2’s stage-1 regime (Qu et al. [28]): their graph structural-causal-model prior, their envelope of shapes and class counts, and their 500k updates. Stages B and C draw each update’s batch from that prior or, with equal probability, from self-supervised episodes over the corpus of real tables used by TabDPT (Ma et al. [23]), while widening the tables and taking the support from a fixed 1,024 rows to a maximum of 32,768. Section F describes both arms in more detail.

StagePriorRowsFeaturesUpdatesPeak LRGPU-h
ASynthetic only1,0241–100500k8×10−48 \times10^{-4}1,600
BSynthetic and real; 50% – 50%256–16,3844–128200k8×10−58 \times10^{-5}540
CSynthetic and real; 50% – 50%256–32,7684–128130k1×10−51 \times10^{-5}288

Table 2. The pretraining curriculum of FTFM

At inference time, we ensemble FTFM by perturbing the sign of numerical features, randomly perturbing the categorical codes, and randomly applying power transformations to features. We do not shuffle the feature order since our method does not use RoPE and is fully permutation invariant. We create an ensemble of 8 base models, each fed with a perturbed variant of the input table.

Optimization. We use the same training pipeline for all stages, apart from the prior hyperparameters described in Table 2. As an optimizer, we deploy Muon (Liu et al. [21]) on every matrix parameter, with momentum 0.9, Nesterov acceleration, five Newton–Schulz steps, decoupled weight decay 0.01, gradients clipped at 10 and non-finite ones skipped. The schedule is a linear warmup over the first 1% of updates and a cosine decay to 18×10−4\frac{1}{8 \times10^{-4}} of the peak in stage A and 18×10−3\frac{1}{8 \times10^{-3}} in stages B and C. Batch size is not fixed: each update packs tables until an activation-memory budget is met, capped at 48 tables in stage A and 128 afterwards, with table shapes synchronized across data-parallel ranks. Stages B and C keep an exponential moving average of the weights at 0.999, and it is that average which is released and evaluated.

Evaluation and baselines. We study classification in this paper as a first experimental framework for verifying the feasibility of factorized table models. We evaluate on two benchmarks, both of which pin their datasets, splits, and metrics: TabArena (Erickson et al. [6]), whose classification suite is 38 datasets, and BeyondArena (Purucker et al. [26]), which adds non-IID, temporal, grouped, and size-stressed slices over 98 classification datasets and on which foundation models have not been leading so far.

As baselines, we take the strongest representative of each architecture family, and they span two orders of magnitude in size. TabFM (TabFM Team, 2026) is the largest at 1.64B parameters, a collapsing zero-shot model released as code and weights with no technical report, so our account of its architecture comes from the released implementation. TabICLv2 (Qu et al. [28]), the strongest fully open model and a two-stage collapsing architecture, is 27.6M, of which 26.3M sit in a deep stack that runs after the feature axis has been collapsed away. EXAONE Tabular (Lee et al. (2026)) is smaller still at 20.81M and is feature-space preserving. TabPFN-3 (Grinsztajn et al. [10]) is the strongest of the TabPFN line, a collapsing architecture having 53.2M parameters that scales to 10610^{6} training rows and adds test-time compute scaling. Tuned gradient-boosted trees (Chen & Guestrin, [3]; Prokhorenkova et al., [25]) serve as the classical reference.

Research Questions (RQs) and Associated Experiments

RQ1: Are factorized tabular models expressive enough?

Since FTFM is a novel paradigm for tabular foundation models that does not apply any direct cell-to-cell attention, one might suspect that it introduces a restrictive expressivity bias. To investigate this hypothesis, we test the ability of FTFM to fit a prior in a controlled experiment. We meta-learn FTFM following the Stage-A curriculum specified in Table 2, which is the exact prior and number of update steps that TabICLv2 was pretrained on. If FTFM outperforms TabICLv2 on its own prior and meta-learning regime of Stage-A, this indicates FTFM does not have obviously restrictive model expressivity.

The empirical evidence presented in Figure 1 demonstrates that FTFM fits the same prior better than the TabICLv2 model, at the same budget. In addition, it shows that on the prior FTFM is learned on, our model outperforms EXAONE Tabular and TabPFN-3 by a statistically significant margin, and is second only to the 10×10\times-larger TabFM model. We deduce that our modeling of tables does not have an expressivity bias.

Figure 1. Performance on 4K held-out tables from stage 1 of the TabICLv2 prior, with every model scored on the same tables. (a) Average ranks; a bar indicates a non-significant difference following a Wilcoxon signed-rank test; (b) mean paired Δ\DeltaCE (cross-entropy) against FTFM; (c) analysis by number of features and number of classes.

RQ2: Do the meta-learning stages lead to progressive gains?

As a second question, we validate whether the choice and sequence of the three pretraining stages of Table 2 lead to any meaningful benefit. To test this question, we first evaluate each stage’s checkpoint on the TabArena Lite version (Erickson et al. [6]).

All checkpoints are under one serving policy so that only the pretraining differs. None of the TabArena or BeyondArena datasets occur in any pretraining datasets. The results on TabArena Lite are shown in Table 3. We notice that the larger tables and the real datasets of stage B give a tangible lift in performance. Ultimately, the checkpoint of stage C, which is trained for a longer budget and on bigger tables, results in better out-of-distribution performance.

StageMean errorC wins onp(C vs.)p(C\ \text{vs.})
A0.169230/38< 0.001
B0.162024/380.030
C0.1614——

Table 3. The performance of each pretraining stage evaluated out-of-distribution on TabArena Lite. Significance is a two-sided Wilcoxon signed-rank test.

Furthermore, we check the in-distribution performance of the per-stage checkpoints, evaluating all stages on the specific prior of each stage. We score all three checkpoints on 2K held-out tables from each of the three stage priors in turn. The results are presented in Figure 2. We observe that meta-learning on the bigger tables of stages B and C non-significantly deteriorates the performance on the small table prior of Stage A. However, the later stages intuitively give a lift on the larger tables they are meta-learned from.

Figure 2. Cross in-distribution evaluation of the pretraining stages.

RQ3: Can FTFM reach the state of the art?

We highlight that the actual empirical research in the field of tabular foundation models compares models using benchmark performances. However, the considered methods are pretrained on different priors and with different training budgets, use different feature preprocessing of the datasets and different feature representation layers in the architectures, have different capacities, and are even ensembled differently at inference time. This practice makes it hard to assess whether the benefit of a released model comes from the architecture, especially if the pretraining code and data are not released for ablating the components. We also follow such performance-oriented practice in this paper, since ablating the benefit of our architecture would demand retraining the baseline models on our prior at the same budget and with the same feature preprocessing and ensembling, which is infeasible within our available computational budget.

Nevertheless, a positive standard in the field of tabular foundation models is the streamlined benchmarks, the most popular of which are TabArena and BeyondArena. In this section, we present the empirical performance of our method on these two benchmarks.

Results on the BeyondArena benchmark

In BeyondArena, our method achieves state-of-the-art performance as indicated in Figure 3. FTFM is first of 29 methods: ahead of TabPFN-3, TabICLv2 and TabPFN-2.6 on Elo and on mean rank, and lowest of every entrant on improvability. Every row in that figure, ours and the published ones alike, is scored by the benchmark’s own context on exactly the datasets and splits we ran. Furthermore, we detail the performance across the types of datasets in Table 4.

Figure 3. Performance of FTFM on the 98 classification datasets of the BeyondArena benchmark. FTFM leads TabPFN-3 and TabICLv2 on Elo and on mean rank, and is lowest on improvability, which is the margin by which a method is beaten. The ranks are computed against 29 methods covered in the benchmark paper.

SlicennTabPFN-3 (default)TabICLv2 (default)RealMLP (t+e)CatBoost (t+e)FTFM
tiny391366 (2)1333 (3)1287 (5)1233 (9)1426 (1)
small261408 (2)1323 (4)1270 (5)1196 (7)1510 (1)
medium201201 (2)1121 (10)1201 (1)1180 (3)1168 (4)
large13912 (16)668 (21)1285 (1)1237 (5)772 (17)
low-dim661334 (2)1312 (3)1261 (5)1222 (6)1390 (1)
high-dim321139 (4)983 (17)1212 (1)1140 (3)1095 (11)
high-cardinality211068 (7)937 (14)1146 (4)1179 (1)930 (15)
random771341 (2)1273 (3)1251 (5)1202 (7)1361 (1)
grouped111055 (22)1097 (16)1279 (2)1215 (5)1206 (7)
temporal101051 (9)755 (19)1184 (1)1145 (4)866 (17)
text91200 (3)1064 (13)1286 (1)1217 (2)998 (18)
all981264 (2)1195 (6)1241 (3)1192 (7)1280 (1)

Table 4. BeyondArena Elo by slice on the classification datasets of each slice that our run covers, with the place out of 29 methods in parentheses and nn the number of datasets. Every column is recomputed on those same tasks with the benchmark’s own evaluator, so a row compares five methods on one set of datasets.

The results of Table 4 indicate that FTFM is the best method on tiny, small, and low-dim datasets, as well as those having random effects, while overall leading with an Elo of 1280. This matches the expectations since our prior does not have datasets with high-cardinality, grouped features, or temporal aspects. Overall, we deduce that FTFM offers a novel architecture that can be trained to the performance standard of the state-of-the-art closed and open tabular foundation models, and clearly outperforms traditional classifiers such as RealMLP or CatBoost.

Results on the TabArena benchmark

We also evaluate FTFM on the TabArena benchmark, a living benchmark consisting of 38 classification datasets. The rank comparisons of the leading methods are presented in Figure 4. In the appendix, we also provide a leaderboard plot in Figure 7 and the detailed results in Table 5. Overall, FTFM rivals the top foundation models and ranks among the top 3 methods out of 72 baselines evaluated in TabArena.

Figure 4. Critical-difference diagram over the full TabArena protocol, 38 datasets. Average rank, lower is better; a bar joins methods whose performance differences are not statistically significant. In contrast to Figure 7, here the ranks are computed only among the five selected methods.

Figure 7. TabArena classification on 38 datasets and 594 splits under the full protocol. Results are recomputed with TabArena’s evaluator from published per-split results. Ranks are computed against the benchmark’s 72 published methods, with selected methods shown here.

Table 5: Error on the 38 TabArena classification datasets under the full protocol, averaged over splits. The metric is 1−AUC1 - \mathrm{AUC} for binary tasks and log-loss for multiclass tasks. Lower is better, and the best result in each row is bold. Baselines use published per-split results.

DatasetTabICLv2TabPFN-3EXAONETabFMFTFM
APSFailure0.00580.00610.00610.00540.0062
Amazon_employee_access0.14690.14990.12850.14200.1398
Bank_Customer_Churn0.12810.12480.12530.12330.1253
Bioresponse0.13820.12280.11850.11360.1237
Diabetes130US0.33660.32960.31860.32990.3327
E-CommerceShippingData0.25490.25290.25320.25740.2532
Fitness_Club0.17920.17850.17940.17880.1790
GiveMeSomeCredit0.13290.13160.13160.13120.1325
HR_Analytics_Job_Change_of_Data_Scientists0.19380.19390.19300.19330.1937
Is-this-a-good-customer0.25190.24870.24310.24640.2461
MIC0.44540.44490.42480.43060.4275
Marketing_Campaign0.07520.08380.07910.07360.0752
NATICUSdroid0.01220.01250.01230.01140.0123
SDSS170.07140.06410.06820.06930.0723
anneal0.02010.01590.01080.01320.0143
bank-marketing0.23690.23340.23240.23060.2332
blood-transfusion-service-center0.24460.24600.25040.24410.2434
churn0.06480.06620.06590.06660.0578
coil2000_insurance_policies0.22640.22750.21730.21220.2234
credit-g0.20360.20660.19640.19420.1955
credit_card_clients_default0.20890.20980.20930.20690.2077
customer_satisfaction_in_airline0.00480.00460.00430.00380.0050
diabetes0.16080.15910.16030.15790.1578
hazelnut-spread-contaminant-detection0.00510.00540.00640.00230.0048
heloc0.19850.19770.19750.19740.1973
hiva_agnostic0.19860.17990.17460.18910.1852
in_vehicle_coupon_recommendation0.15620.15420.15500.15260.1488
jm10.22640.22750.21770.21060.2205
kddcup09_appetency0.16980.17500.15010.15490.1664
maternal_health_risk0.39720.40370.41460.37060.3857
online_shoppers_intention0.06250.06140.06110.06010.0611
polish_companies_bankruptcy0.01560.01500.00950.00470.0071
qsar-biodeg0.05910.06100.06160.05800.0591
seismic-bumps0.21780.21560.21420.20610.2123
splice0.09970.10600.09880.09610.0956
students_dropout_and_academic_success0.53350.53970.51750.50780.5255
taiwanese_bankruptcy_prediction0.05370.05190.05260.04910.0506
website_phishing0.22260.21730.21640.21010.2227
Mean0.16740.16640.16250.16070.1632
Mean rank4.053.662.761.792.74

Table 5.

We further highlight that the empirical performance of tabular foundation models depends on the quality of the inference-time ensembling. We conducted an experiment on the TabArena Lite benchmark, where we first ran an apples-to-apples experiment where each model is run without ensembling by receiving a single zz-normalized table. Then we compare in an apples-to-oranges manner the ensembled variants of the methods (each with a different ensembling strategy) and present the results in Figure 5. FTFM is not worse than TabFM with a statistically significant margin on the apples-to-apples experiment, indicating that that part of the reported performance of models comes from their efficient ensembling.

Figure 5. Average rank over the 38 TabArena Lite datasets, apples-to-apples (left) and apples-to-oranges (right); each line is one method moving between the regimes. On the right, every method is bagged over 8 members, but each brings its own different ensembling pipeline. The bracket carries the two-sided Wilcoxon signed-rank test of FTFM against TabFM, Holm-corrected over the ten pairs. The left column shows our runs and the right takes the four baselines from the benchmark’s published values on identical dataset splits.

Inference costs

As a last remark, in § 3.3 we hypothesized that FTFM has the asymptotic inference costs of feature-collapsing baselines, even though it preserves the feature space. To validate this claim, we present in Figure 6 the prediction times of our method compared to TabICLv2 and TabPFN-3 on the 98 datasets of the BeyondArena benchmark. We plot the number of tokens in each table versus the prediction time in seconds. As observed, our method shows a linear log-log relationship between inference cost and data size, comparable to feature-collapsing methods. We present the full fit and prediction results for all the methods in Table 8 in the appendix.

Figure 6. Prediction time against table size on the BeyondArena classification datasets.

DatasetFTFMTabPFN-3TabICLv2RealMLPCatBoost
fitpred.fitpred.fitpred.fitpred.fitpred.
acquire_valued_shoppers_challenge-f364351.045459.934435.48032819.550642.2
amazon_employee_access-4417c4c952b2085.8454.7202.3959424.51526.3
amex_non_iid_1m-e2e64ee8b42e205961320.8200495953.45329839.0278841766.385907109.0
aves_voting_2026-58a0e941922e259238.732023.336027.49052134.71029329.0
aps_failure-b63fc63301173203137.720431.222922.149991141.541181.9
asp_potasso_classification-6cbac33e35213.4221.1261.5213740.22540.5
audiology_diagnosis-90d4820bace0630.4571.1250.6371820.47371.1
bad_customer_detection-7b2ea3462dc250.5251.1141.0165918.12741.1
bank_customer_churn-b1fc1f13ea36711.5311.7130.9374911.63520.4
bank_marketing-ba2b45409a5132910.7627.1283.51596423.16972.3
biogeographical_ancestry_prediction-960.8811.5480.7341142.8411510.0
biomechanical_orthopaedic_prediction650.3511.0210.534498.510040.3
biresponse-63f85b558d7087533.31256.620511.810586344.239817.5
blood_transfusion-c08bfb0f04f3680.4521.1210.6325918.15070.3
cardiotocography-172ef52f9a1b320.6251.5120.618818.84430.5
churn-4ae7db677764491.0321.5110.7292715.45610.6
clock_protein_toxicity-f1eed69eb7842151.2911.41280.96919293.140614.4
coil_2000-d498c6d7849f1504.8463.3241.6489021.94041.7
consumer_complaints_1m-853af219eb5a75011045.4194443250.3230703766.8304315297.7322811110.3
covertype-1e1c3d238643065661.6381486.6416359.1371678.26481.5
credit_approval-a37d9105992b710.4531.1230.6364131.75881.6
credit_card_clients_default-9470ea92367.0404.3242.4920217.44911.1
credit_g-6ced9fa4ef3f280.4221.1120.7155519.43031.3
customer_satisfaction_in_airline-71138165.120737.011918.44561228.638204.3
dementia_prediction-d8ce15bd8d63590.3481.0210.6342114.16730.2
diabetes_130_us-06cb03600f076636.312216.0738.92035944.528138.6
drug_induced_autoimmunity_predictio1120.9581.2410.7406860.92071.2
early_stage_diabetes_risk_predicti590.4511.1210.6348819.79010.6
ecoli_proteins-88ead48fb030640.3461.0220.6363715.76590.3
DatasetFTFM fitFTFM pred.TabPFN-3 fitTabPFN-3 pred.TabICLv2 fitTabICLv2 pred.RealMLP fitRealMLP pred.CatBoost fitCatBoost pred.
ecommerce_shipping-db33e797abb9721.6261.7120.9430818.82870.4
emscad-1790bb44ad9161426.110111.89911.21492178.3632010.3
eryhemato_squamous_disease-a64cf6ea9690.4621.2300.634795.860473.0
fitness_club-95daf6614f7a290.4231.190.7158712.22560.4
forensic_glass_identification-e30419600.4451.0200.5349010.813870.4
gallstone_disease-63982cfe3e7c630.3491.1250.6347534.98130.8
ghanas_indigenous_intel-ecbbda50d44e851.3311.8231.1456012.325060.7
give_me_some_credit-51f4a5f15bd6152668.125346.212522.05498135.511752.9
hazelnut_spread_contaminant_detecti380.7191.6130.7216010.310930.9
heart_disease_cleveland-1e633334e0ef630.3501.0220.6361023.56481.2
heart_disease_hungary-f5ca5cfb8352670.3531.1230.6350524.86211.4
heart_disease_va_long_beach-062b5d630.3491.1220.6346930.05971.6
heart_failure_followup_survival-25d680.3481.1220.6343728.66220.6
heloc-d2ac12c6b994872.2261.9151.0366912.23630.6
hepatitis_c_prediction-d17a9e02b43b680.4471.1220.6349013.210060.3
hepatitis_survival_prediction-d16737610.3501.1230.6351933.75781.0
hiva_agnostic-c4b35582c20470827.51256.516510.93751544.818824.5
home_credit_default_risk-1b1ae7eb0511501199.41992267.0128772688.5236089858.68639070.1
home_credit_default_stability_1m-e38863714.7240195980.76072854.03215642054.3239805125.9
homeq_default_prediction-6da1464558b571.2221.4100.7369413.05020.8
homesite_quote_conversion-f57214c4e01908779.71363201.32017286.9203535291.94378832.8
horse_colic_survival-e1beb13d0b4e680.3521.2240.6336320.84871.6
hotel_booking_demand-9541da6f5a2b112815.116918.41089.93474723.659552.9
hr_analytics-9f0cb22a2cd41233.1312.7171.4599919.14391.7
ieee_fraud_detection-5e0af5cbbb7319714196.45279680.9305174448.8282144880.1175089190.0
in_vehicle_coupon_recommendation-101082.8312.2161.2579426.310272.8
indian_liver_patient_dataset-0216cd640.4471.0210.6343832.74620.3
iranian_churn-87d6fa636729390.6201.4120.624458.13420.5
jm1-c8cfe392b650972.4251.9151.0516220.06020.6
kdd_cup_09_appetency-72d0143e7c3d226898.319221.920519.722486171.6408210.2
kick-57821d387256115513.216317.6979.42833120.045965.7
kickstarter-fc5123d788a6641483.973493.567161.69865917.6442498.1
labour_inspection_compliance-17b501f6227269.129532.844846.371565410.61076322.7
lending_club_1m-f6bf13e264e49278358.9148562447.15934641.4301036458.5149009111.4
ljubijana_breast_cancer-28c8b88c6b23650.3501.1210.6366528.74311.0
ljubijana_primary_tumor-b1acf2d83b0d1981.1511.1320.9353627.97460.8
lung_cancer-48c9b1f8934d297112.957277.7251110.33752514.92052214.9
lung_cancer_epithelial_genexp-ff623717423.51931614.2757022.275574734.118732425.2
marketing_campaign-45b76b0f0de6360.7291.7130.8192720.33271.1
maternal_health_risk-02da4815804270.4190.9110.614987.43100.4
mic-ba10980b94f8551.4331.6210.9175333.129842.8
mice_protein_trisomy_discriminant-11321.1521.1390.8427925.6128780.7
micro_mass-5deb16d4fa0e218824.21041.41732.41087613.21050525.5
musk-16dca0dbddf55647.21595.0902.41244129.367111.0
naticusdroid_android_permissions_da1183.8403.2211.4314333.85300.9
online_shoppers_purchasing_intentio902.2372.0131.1461017.44361.4
otto_group_product_classification_101140.710916.59910.52808634.7303484.2
pancreatic_cancer_mouse_detection-b12646.617785.09463.915044600.7343159.3
parkinsons_biomedical_voice_measure630.4501.1280.6378022.219580.5
polish_companies_bankruptcy-c6e7be381033.1302.2151.0347526.48720.5
porto_seguro-7894967113f319761128.73208651.51607308.4194355128.81132232.1
predict_students_dropout_and_acade531.3281.8160.8251314.96771.7
prostate_cancer_detection-d59b58af23446026.080259.1340919.74233742.619863914.8
pva_revenue_prediction_kddcup98-94a11834182.61270122.8204514257.7233580879.83268580.9
qsar_biodeg-99deb20f66bb270.5211.1140.7158411.33840.7
regensburg_pediatric_appendicitis-e7381.0271.3180.8181438.73582.3
santander_customer_satisfaction-5042231897.822231.627226.64249566.925242.4
santander_customer_transaction_pred2154388.31011139.32910445.9123073675.06367776.0
sdss-17-3d9b1f6de2a160723.29515.5497.6188349.2861510.0
seismic_bumps-9636045db12f320.6211.4120.7182016.22630.6
sepsis_prediction_1m-77a236a7d8eb26806999.5170133021.896551663.1285663271.5764316.6
sepsis_survival_minimal_clinical_r85435.3153527.76712.03506422.94060.4
south_africa_coronary_heart_disease610.3451.0210.6365528.75210.4
splice-0be1e96684a9531.5382.1200.8232622.914733.4
taiwanese_bankruptcy_prediction-e1d11163.5382.8221.3370718.58230.8
thyroid_discordant-1111dc4e4f7d441.0291.6120.7200813.74540.9
tour_travels_churn-638a10e1adbe260.4201.1110.6208411.82810.3
website_phishing-e1ffd9e6d3a250.4211.1140.719847.72560.6
wids_diabetes_mellitus-d05f898265f26447279.050672.048951.665939212.32099613.6
Median1102.2511.9241.0392424.010161.5

Table 8. Fit and prediction times in seconds, averaged over splits. FTFM uses one H100, while baselines use the benchmark maintainers’ hardware, so absolute times are not directly comparable. Figure 6 compares scaling with table size, which a constant hardware factor preserves. † marks FTFM’s two reduced-support results (rung 3).

These findings show that our novel factorized table modeling can achieve state-of-the-art performance while also offering practical benefits to practitioners by matching the speed of the fastest feature-collapsing tabular foundation models.

Limitations

To validate the novel direction of factorized table representations, we focused on the classification task, leaving regression as future work. Furthermore, our architecture can benefit from the recent Fourier feature representation introduced in TabFM Team (2026).

Conclusion

We introduced FTFM, a tabular foundation model in which a cell representation is a factorization of one contextual row latent against one column latent rather than the product of cell-to-cell attention. Our method achieves strong performance on the BeyondArena and TabArena benchmarks, outperforming state-of-the-art methods in the field, such as TabPFN-3 and TabICLv2. We showcased that FTFM, a novel architecture based on table factorization, achieves competitive performance in modern tabular representation learning and can serve as the future architectural blueprint for efficiently modeling tables.

Reproducibility Statement

The architecture is specified operation by operation in Section 3.2. The overall design choices and hyperparameters are detailed in the paper. Our evaluation uses the published fixed protocols of TabArena and BeyondArena, which pin datasets, splits, and metrics. The synthetic pretraining prior follows the structural-causal-model family of Qu et al. [27, 28] and the datasets of Ma et al. [23]. Inference code and a link to the model weights accompany the submission.

AI Use Statement

We used generative AI to prototype the source code for the experiments, followed by manual inspection and testing of the generated code. We also used AI to check for grammar issues and rephrase the manuscript text. Furthermore, generative AI models were used to generate and polish the visual aspects of the figures and tables. We did not use AI to generate the core idea, suggest experiments, baselines, datasets, or write the proposed method. We reviewed all AI-assisted work.

References

Full-protocol TabArena leaderboard

Figure 7 complements the BeyondArena results in Figure 3, using the same plot format.

Per-dataset TabArena results

Table 5 reports the per-dataset results behind Figure 7, using published baseline results.

Results across pretraining stages

Table 6 reports the values in Figure 2. All checkpoints are evaluated on the same 2,000 held-out tables within each prior. Compare checkpoints within rows because the priors differ in difficulty.

Table 6: Mean cross-entropy and average rank by checkpoint and evaluation prior. The best value for each metric within each prior is bold. On the Stage-B prior, different checkpoints minimize mean error and rank. The lower panel gives two-sided Wilcoxon signed-rank pp-values, Holm-corrected across the three pairs within each prior.

Scored onStage-AStage-BStage-C
CErankCErankCErank
Stage-A0.28661.920.28832.090.28781.98
Stage-B0.35662.400.32781.850.32861.75
Stage-C0.40962.480.39001.860.39001.66
Scored onA vs BA vs CB vs C
Stage-A6.9×10−86.9 \times10^{-8}0.0012.3×10−52.3 \times10^{-5}
Stage-B1.0×10−1021.0 \times10^{-102}2.8×10−1132.8 \times10^{-113}0.033
Stage-C1.1×10−1261.1 \times10^{-126}3.7×10−1433.7 \times10^{-143}8.4×10−148.4 \times10^{-14}

Table 6.

Per-dataset BeyondArena results

Tables 7 and 8 report errors and runtimes for all 98 BeyondArena classification datasets. FTFM timings use one H100, while baselines use the benchmark’s hardware, so the times are not directly comparable.

DatasetFTFMTabPFN-3TabICLv2RealMLPCatBoost
acquire_valued_shoppers_challenge-f0.35180.34680.36810.35160.3559
amazon_employee_access-4417c4c952bc0.13180.14890.14190.12760.1134
amex_non_iid_1m-e2e64ee8b42e0.05740.06180.06310.05460.0559
anes_voting_2026-58a0e941922e0.12900.12000.13050.12830.1272
aps_failure-b63fc63301170.00670.00600.00620.00830.0086
asp_potassco_classification-6cbac33e2.21392.20672.19932.21582.2245
audiology_diagnosis-90d4820bace00.29020.29370.25580.27490.3323
bad_customer_detection-7b2ea3462dce0.24740.25090.25360.26090.2496
bank_customer_churn-b1fc11f3ea360.12500.12360.12900.12870.1290
bank_marketing-ba2b45409a510.23490.23360.23480.23770.2351
biogeographical_ancestry_prediction-1.33881.37261.34131.40561.4878
biomechanical_orthopaedic_prediction0.34970.34950.35110.35220.3552
bioresponse-63f85b558d700.12930.12630.14370.12590.1307
blood_transfusion-c0b8fbb104f30.25010.24960.25280.25590.2599
cardiotocography-172ef52f9a1b0.21880.24900.23430.22880.2146
churn-4ae7db6777640.06180.06960.06320.07850.0776
clock_protein_toxicity-f1eed69eb7840.48230.49890.51690.48580.4915
coil_2000-d498c6d7849f0.21910.22000.21990.23640.2330
consumer_complaints_1m-853af219eb5a0.50720.41040.44520.43530.4251
covertype-1ed1c3d238641.05880.96751.02530.71150.7172
credit_approval-a37d9105992b0.06220.06650.06380.06820.0670
credit_card_clients_default-9470ea90.21120.21260.21160.21750.2191
credit_g-6ced9fa4ef3f0.19330.20050.20080.20250.2008
customer_satisfaction_in_airline-710.00510.00460.00480.00500.0053
dementia_prediction-d8ce15bd8d630.67850.71900.69370.61360.6355
diabetes_130_us-a06cb03600f00.33550.33310.33920.33210.3290
drug_induced_autoimmunity_predictio0.14450.16300.15190.17390.1811
early_stage_diabetes_risk_predicti0.02350.02730.02410.03660.0243
ecoli_proteins-88ead48fb0300.35120.36260.35730.36130.3666
ecommerce_shipping-db33e797abb90.25240.25340.25490.25510.2562
emscad-1790bb44ad910.00470.00810.00600.00730.0120
eryhemato_squamous_disease-a64cf6ea90.04480.06120.05430.06980.0661
fitness_club-95daf6614f7a0.17990.17920.17950.18180.1853
forensic_glass_identification-e304190.64170.68750.68950.77180.7654
gallstone_disease-63982cfe3e7c0.09070.07980.08580.10860.1395
ghanas_indigenous_intel-ecbbda50d44e0.18390.19880.24280.17390.1893
give_me_some_credit-51f4a51f5bd60.15160.13630.15110.15180.1537
hazelnut_spread_contaminant_detecti0.00270.00360.00250.01410.0274
heart_disease_cleveland-1e63334e0ef0.09600.09350.09350.09470.0943
heart_disease_hungary-f5ca5cfb83520.10190.09250.09830.09750.1075
heart_disease_va_long_beach-062b5d0.32270.32650.33010.35730.3183
heart_failure_followup_survival-25d0.07840.09210.08510.09580.0893
heloc-d2ac12c6b9940.20720.20740.20740.20770.2109
hepatitis_c_prediction-d17a9e02b43b0.09980.10480.11560.12370.1425
hepatitis_survival_prediction-d167370.11810.13580.12720.14420.1331
hiva_agnostic-c4b35582c2040.21020.21710.24910.22110.2164
home_credit_default_risk-1b1ae7eb050.23470.21340.22230.20760.2079
home_credit_default_stability_1m-e0.15930.14420.16390.12790.1310
homeq_default_prediction-6da146558b0.00000.00010.00120.00740.0220
homesite_quote_conversion-f57214c4e00.03600.03600.04130.03220.0318
horse_colic_survival-e1beb13d0b4e0.68130.70290.70580.69970.7282
hotel_booking_demand-9541da6f5a2b0.20730.22890.24390.18260.1703
hr_analytics-9f0cb22a2cd40.19520.19600.19580.19780.1966
ieee_fraud_detection-5e0af5cbb730.12120.10080.10960.08270.0870
in_vehicle_coupon_recommendation-100.15290.15830.16190.15580.1577
indian_liver_patient_dataset-0216cd0.23130.23620.22850.24650.2475
iranian_churn-87d6fa6367290.00260.00270.00300.00460.0096
jm1-c8fce392b6500.22380.22900.23090.24950.2468
kdd_cup_09_appetency-72d0143e7c3d0.15550.16490.15560.17320.1428
kick-57821d3872560.20490.20490.20580.20700.2076
kickstarter-fc5123d788a60.07550.06940.07650.06790.0685
labour_inspection_compliance-17b501f0.24480.20830.20840.21050.2156
lending_club_1m-f6bf13e264e40.3047†0.29620.31480.28510.2850
ljubijana_breast_cancer-28c8b88c6b230.27320.27890.27820.31060.3046
DatasetFTFMTabPFN-3TabICLv2RealMLPCatBoost
ljubljana_primary_tumor-b1acf2d83b0d1.30921.32161.33281.31751.3763
lung_cancer-48c9b1f8934d0.22250.12690.19130.14590.2041
lung_cancer_epithelial_genexp-ff6230.20250.24260.27730.24820.2536
marketing_campaign-45b76b0f0de60.08690.09360.08690.10210.1083
maternal_health_risk-002da48158040.30270.29710.29940.35670.4010
mic-ba10980b94f80.42750.44240.44160.43750.4642
mice_protein_trisomy_discriminant-10.78851.16420.92191.21921.3395
micro_mass-5deb16d4fa0e0.39480.69370.52500.62030.5135
musk-16dca0dbddf50.15670.12810.10900.12260.1926
naticusdroid_android_permissions_da0.01170.01250.01230.01320.0130
online_shoppers_purchasing_intentio0.05590.05560.05680.06060.0607
otto_group_product_classification_0.46810.43920.41750.43740.4611
pancreatic_cancer_mouse_detection-b0.32550.36450.49400.44420.4648
parkinsons_biomedical_voice_measure0.19020.24320.17000.12560.1507
polish_companies_bankruptcy-c6e7be380.00410.00900.01050.02910.0310
porto_seguro-7894967113f30.36380.35820.36540.35750.3571
predict_students_dropout_and_acade0.52060.53580.52870.53410.5369
prostate_cancer_detection-d59b58af230.00380.00720.06060.01710.0179
pva_revenue_prediction_kddcup98-94a0.42180.38380.39510.37310.3689
qsar_biodeg-99edbe20f66b0.05640.05770.05640.06150.0641
regensburg_pediatric_appendicitis-e70.06340.06280.06440.07720.0776
santander_customer_satisfaction-50420.15130.16440.18620.15470.1554
santander_customer_transaction_pred0.12470.11830.11570.10210.0921
sdss_17-3d9b16fdea210.06860.06010.06660.08350.0708
seismic_bumps-9636045db12f0.20800.21150.21460.22650.2194
sepsis_prediction_1m-77a236a7d8eb0.18200.22550.20130.18210.1877
sepsis_survival_minimal_clinical_r0.29330.29330.29800.29320.2933
south_africa_coronary_heart_diseas0.23100.22810.22760.23940.2437
splice-0b61e96684a90.10940.11670.11370.12440.1283
taiwanese_bankruptcy_prediction-e1d10.04830.04840.05090.05230.0531
thyroid_discordant-1111dc4e4f7d0.01110.00920.00910.01260.0204
tour_travels_churn-638a10e1adbe0.03950.03800.04170.04470.0420
website_phishing-e1ffdd9e6d3a0.25550.24430.25000.26540.2734
wids_diabetes_mellitus-d05f898265f20.13790.14150.14140.12160.1260
Mean0.25450.26210.26200.26170.2672
Mean rank2.282.683.063.293.69

Table 7. Error on the 98 BeyondArena classification datasets, averaged over splits. Lower is better, and the best result in each row is bold. Baselines use published per-split results. † marks FTFM’s two reduced-support results (500,000 rows, rung 3), which are not directly comparable to full-support results (see Section E).

Serving policy on BeyondArena

BeyondArena fixes datasets, splits, preprocessing, bagging, and metrics. Our serving settings follow Table 9, and we record the lowest feasible rung for each dataset. Table shape determines rungs 0 and 1. Rungs 2 and 3 apply only after a failure at the preceding rung. The released code identifies the affected datasets.

RungServingAttentionMembersBudgetSupportnn
0fp32fp3284 hfull88
1fp32fp32130,600 sfull2
2fp32fp32180,000 sfull6
3fp32fp32180,000 s500k rows2

Table 9. Serving settings for the 98 BeyondArena classification datasets. TT and MM count support rows and columns after preprocessing, with work = T2MT^{2}M and cells = TMTM. All rungs use fp32, including attention. Higher rungs reduce ensemble size, increase the time budget, then reduce support.

Precision. We retain fp32 for consistent numerics. FlashAttention has no fp32 kernel, so we use the memory-efficient kernel, whose T2T^{2} cost can exhaust a fold’s budget. The largest completed table had work = 6.894×10116.894 \times10^{11}. The smallest table that failed had 8.999×10118.999 \times10^{11} and took 2,474 s for one fold.

Escalation. Rung 1 applies when work >1.2×1013> 1.2 \times10^{13} or cells >4.3×107> 4.3 \times10^{7}. The first threshold comes from the largest table that completed with eight members (744 s per fold). Above the second, the cell grid exceeds accelerator memory and prefill streams from host memory, about 4.5×4.5 \times slower. Reducing the ensemble costs about 1% in performance. TabICLv2 makes the same reduction above 700k rows. For the six datasets that timed out at rung 1, we increased only the budget at rung 2. At 13,324 s and 14,772 s per fold, nine sequential fits exceeded available budgets for two single-split datasets. Rung 3 caps their support at 500,000 rows, down from 814,751 and 1,000,000 for the scored refit (712,907 and 875,000 per bag fold). We retain one row per class and sample the rest uniformly with a fixed seed. Both results are marked as using reduced support. Omitting them would trigger leaderboard imputation.

Bagging across jobs. BeyondArena assigns one budget to the whole bagged fit, and AutoGluon allocates about limit⁡/(folds⁡+1.2)\operatorname{limit}/(\operatorname{folds} + 1.2) per fold. A large fp32 fold can exceed this share (20,549 s versus 8,889 s on amex_non_iid_1m), triggering AutoGluon’s deadline check. A sufficient budget would require about 50 h in one job. We therefore run folds separately and merge the results. Out-of-fold predictions sum exactly because each fold’s array is zero outside its validation rows. Seeding folds by absolute index preserves splits and seeds. A test reproduced the single-job result exactly (0.30880966400279025). Only the budget changes, as each fold can use the full limit. We used this procedure for eight datasets, namely amex_non_iid_1m, consumer_complaints_1m, home_credit_default_risk, home_credit_default_stability_1m, ieee_fraud_detection, lending_club_1m, porto_seguro, and pva_revenue_prediction_kddcup98. Of these, only porto_seguro uses rung 0. The other 87 rung 0 datasets run in one job.

Pretraining Priors

This section summarizes the distributions used in § 4 without giving full generation details.

Structural-causal-model prior. The synthetic prior follows Müller et al. [24]; Qu et al. [27, 28]. We propagate noise through a sampled directed acyclic graph with assigned node mappings. Selected nodes supply features, and one supplies the target, discretized for classification. Table shape and labeled fraction vary across draws. We generate tables on the fly without reuse, epochs, or a validation split.

Self-supervised prior over real tables. Stages B and C mix real and synthetic episodes in equal proportions, following Ma et al. [23]; Zhang et al. [37]. We use TabDPT’s curated corpus of public OpenML tables. Removing every table that overlaps with TabArena or BeyondArena reduces the corpus from 123 to 108 tables and keeps evaluation data out of pretraining.

In each episode, we mask one column and use it as the target. The remaining columns are features. We bin continuous targets into classes using random value thresholds and retain the codes of low-cardinality targets. We shuffle and subsample the feature columns to obtain multiple tasks from each table. Rows are sampled uniformly without replacement and split into support and query sets. We use the same loss as for synthetic tables.

We fit feature moments on support rows alone to avoid leaking query statistics into the context. The reference implementation fits them once per table. We also omit retrieval. The upstream implementation builds support from a query row’s nearest neighbours, forming a local cluster with low class diversity. Our serving path reads the full support.

References

  1. [1]Michael Arbel, David Salinas, and Frank Hutter. EquiTabPFN: A target-permutation equivariant prior fitted networks. arXiv preprint arXiv:2502.06684, 2025.arxiv.org/abs/2502.06684
  2. [2]Emmanuel J. Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 9(6):717–772, 2009. arXiv:0805.4471.arxiv.org/abs/0805.4471
  3. [3]Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.DOI
  4. [4]Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Jiang Zhang, Shi Han, and Dongmei Zhang. TabularNet: A neural network architecture for understanding semantic structures of tabular data. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2021. arXiv:2106.03096.DOI
  5. [5]Moonjung Eo, Min-Kook Suh, Hye-Seung Cho, Jiwon Kim, Seoyoon Kim, Sangjun Nam, and Soonyoung Lee. EXAONE Tabular 1.0: Technical report. arXiv preprint arXiv:2608.25774, 2026.arxiv.org/abs/2608.25774
  6. [6]Nick Erickson, Lennart Purucker, Andrej Tschalvez, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data. arXiv preprint arXiv:2506.16791, 2025.arxiv.org/abs/2506.16791
  7. [7]Josh Gardner, Juan C. Perdomo, and Ludwig Schmidt. Large scale transfer learning for tabular data via language modeling. In Advances in Neural Information Processing Systems, 2024. arXiv:2406.12031.arxiv.org/abs/2406.12031
  8. [8]Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on tabular data? Advances in Neural Information Processing Systems, 35, 2022.
  9. [9]Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Lennart Purucker, Noah Hollmann, and Frank Hutter. TabPFN-2.5: Advancing the state of the art in tabular foundation models. arXiv preprint arXiv:2511.08667, 2025.arxiv.org/abs/2511.08667
  10. [10]Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Mihir Manium, Shi Bin Hoo, Magnus Bühler, Anurag Garg, Dominik Safaric, Jake Robertson, Benjamin Jäger, Simone Alessi, Adrian Hayler, Vladyslav Moroshan, Lennart Purucker, Philipp Singer, Alan Arazi, Julien Siems, Jan Hendrik Metzen, Georg Grab, Nick Erickson, Siyuan Guo, Eliott Kalfon, Simon Bing, David Salinas, Clara Cornu, Lilly Charlotte Wehrhahn, Diana Krichkova, Kursat Kaya, Lydia Sidhoum, Marie Salmon, Jerry Chen, Madelon Hulsebos, Yann LeCun, Samuel Müller, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, and Frank Hutter. TabPFN-3: Technical report. arXiv preprint arXiv:2605.13986, 2026.arxiv.org/abs/2605.13986
  11. [11]Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics. In International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017. arXiv:1708.05027.arxiv.org/abs/1708.05027
  12. [12]Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaovi Jiang, and David Sontag. TabLLM: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, 2023. arXiv:2210.10723.arxiv.org/abs/2210.10723
  13. [13]Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019.arxiv.org/abs/1912.12180
  14. [14]Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations, 2023. arXiv:2207.01848.arxiv.org/abs/2207.01848
  15. [15]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637:319–326, 2025.DOI
  16. [16]Rasa Hosseinzadeh, Alex Labach, Zexin Xue, Shuyi Han, Valentin Thomas, and Anthony L. Caterini. TabDPT-Turbo: Efficient in-context learning for tabular prediction. arXiv preprint arXiv:2608.01400, 2026.arxiv.org/abs/2608.01400
  17. [17]Tomoharu Iwata and Atsutoshi Kumagai. Meta-learning from tasks with heterogeneous attribute spaces. In Advances in Neural Information Processing Systems, volume 33, 2020.
  18. [18]Hadi S. Jomaa, Lars Schmidt-Thieme, and Josif Grabocka. Dataset2Vec: Learning dataset meta-features. Data Mining and Knowledge Discovery, 35:964–985, 2021. arXiv:1905.11063.arxiv.org/abs/1905.11063
  19. [19]Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30–37, 2009.DOI
  20. [20]Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International Conference on Machine Learning, 2019.arxiv.org/abs/1810.00825
  21. [21]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, and Junjie Yan. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025.arxiv.org/abs/2502.16982
  22. [22]Lars Lorch, Scott Sussex, Jonas Rothfuss, Andreas Krause, and Bernhard Schölkopf. Amortized inference for causal structure learning. In Advances in Neural Information Processing Systems, 2022. arXiv:2205.12934.arxiv.org/abs/2205.12934
  23. [23]Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L. Caterini. TabDPT: Scaling tabular foundation models on real data. In Advances in Neural Information Processing Systems, 2025. arXiv:2410.18164.arxiv.org/abs/2410.18164
  24. [24]Samuel Müller, Noah Hollmann, Sebastian Pineda-Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. International Conference on Learning Representations, 2022. arXiv:2112.10510.arxiv.org/abs/2112.10510
  25. [25]Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31, 2018.arxiv.org/abs/1706.09516
  26. [26]Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzmüller, Alan Arazi, Alexander Pfefferle, and Mustafa Tajjar. Beyond IID: How general are tabular foundation models, really? arXiv preprint arXiv:2606.30410, 2026.arxiv.org/abs/2606.30410
  27. [27]Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. In International Conference on Machine Learning, 2025. arXiv:2502.05564.arxiv.org/abs/2502.05564
  28. [28]Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. arXiv preprint arXiv:2602.11139, 2026.arxiv.org/abs/2602.11139
  29. [29]Steffen Rendle. Factorization machines. In IEEE International Conference on Data Mining, pp. 995–1000, 2010.DOI
  30. [31]Ruoxi Wang, Rakesh Shivanna, Derek Z. Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. DCN V2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the Web Conference, 2021. arXiv:2008.13535.DOI
  31. [32]Yuanrui Wang, Xingxuan Zhang, Han Yu, Mingchao Hao, Gang Ren, Hao Yuan, Li Mao, Yunjia Zhang, Chun Yuan, and Peng Cui. LimiX-2M: Mitigating low-rank collapse and attention bottlenecks in tabular foundation models. arXiv preprint arXiv:2606.04485, 2026.arxiv.org/abs/2606.04485
  32. [33]Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Z. Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. In International Conference on Learning Representations, 2024. arXiv:2403.01841.arxiv.org/abs/2403.01841
  33. [34]Guri Zabërgja, Rafiq Kamel, Arlind Kadra, Christian M. M. Frey, and Josif Grabocka. End-to-end compression for tabular foundation models. arXiv preprint arXiv:2602.05649, 2026.arxiv.org/abs/2602.05649
  34. [35]Yuchen Zeng, Tuan Dung Kang, and Kangwook Lee. TabFlex: Scaling tabular learning to millions with linear attention. International Conference on Machine Learning, 2025. arXiv:2506.05584.arxiv.org/abs/2506.05584
  35. [36]Xingxuan Zhang, Yuanrui Wang, Han Yu, Mingchao Hao, Gang Ren, and Hao Yuan. LimiX: Unleashing structured-data modeling capability for generalist intelligence. arXiv preprint arXiv:2509.03505, 2025a.arxiv.org/abs/2509.03505
  36. [37]Xiyuan Zhang, Danielle C. Maddix, Junming Yin, Nick Erickson, Abdul Fatir Ansari, Boran Han, Shuai Zhang, Leman Akoglu, Christos Faloutsos, Michael W. Mahoney, Cuixiong Hu, Huzefa Rangwala, George Karypis, and Bernie Wang. Mitra: Mixed synthetic priors for enhancing tabular foundation models. arXiv preprint arXiv:2510.21204, 2025b.arxiv.org/abs/2510.21204

Paper details

vlnc:W_v8qn859c
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI

View statement
Looking for related papers…
Contents