betaFeedback

WorksComputer Science

Attention Quantization for Tabular Foundation Models

Abstract

With the recent rise and adoption of tabular foundation models, optimizing their inference performance becomes an emerging field for efficiency research. While the models are architecturally similar to transformer-based large language models (LLMs), the size and serving patterns differ significantly. We show that the focus should be on the attention calculation and less on weight or KV cache quantization, which are more popular in LLMs. We develop a quantization strategy for queries, keys, and values to FP8 and use explicit FP8 matrix multiplication instructions to speed up the attention calculation. We find that it is crucial to align the quantization error in the test rows with the quantization error in the training rows, as otherwise the accuracy drops drastically. Our Triton kernel achieves a speedup up to 1.7x over regular 16-bit kernels, and we show that on TabPFN-v3 and TabICLv2 there is no relevant accuracy loss across TabArena and BeyondArena.

Introduction

Transformer-based models [Vaswani et al., 2017] are now widely adopted in large language models (LLMs) and recently also in tabular foundation models. Their broad adoption makes it necessary to reduce inference costs and latency. For LLMs, quantization [4, 11, 8] has been one of the most successful approaches to cut costs and latency.

Preprint. For tabular foundation models, such as the TabPFN [6, 5] or TabICL [13] model families, the models themselves are still fairly small, so weight quantization [9] offers limited benefits. We instead explore quantizing the matrix multiplications in the fused attention. This is particularly helpful for tabular foundation models as their cost scales quadratically with the row count of the training dataset, which can be hundreds of thousands or even millions. However, it is unknown how the predictive performance is affected.

Modern hardware accelerators like NVIDIA’s Hopper and Blackwell GPUs, AWS’ Trainium 3, or AMD Instinct MI325X offer instructions for FP8 matrix multiplications that have twice the throughput over their 16-bit counterparts. While speedups can be realized in standalone large matrix multiplications, it is not trivial to realize them in a fused attention kernel, as other operations, like exponentiation, can become the bottleneck [?] . Whether FP8 matrix multiplications can lead to speedups thus depends on the architecture and hardware specifications.

To the best of our knowledge, this is the first study of FP8 attention in the context of tabular foundation models. We use TabPFN-v3 [5] as the workhorse, and our results motivate using models with FP8 attention as a plug-in post-training. First, we show that a per-head absmax-scaled quantization of queries, keys, and values in the in-context learning (ICL) layer of the transformer can be done without relevant accuracy degradation. Remarkably, this is nearly lossless only when both the train–train and the test–train attention are quantized. Second, we show that on an NVIDIA RTX Pro 6000 we can achieve up to 1.7x kernel speedups through FP8. While our ablations are centered around one model, we also evaluate on TabICLv2 [13] and obtain similar accuracy and speed results.

Method

A softmax attention head maps queries Q∈RSq×dQ \in\mathbb{R}^{S_q \times d}, keys K∈RSkv×dK \in\mathbb{R}^{S_{kv} \times d}, and values V∈RSkv×dV \in\mathbb{R}^{S_{kv} \times d} to

Attn⁡(Q,K,V)=PV,with P=softmax⁡(αQKT)(1)\operatorname{Attn}(Q, K, V) = PV, \qquad\text{with } P = \operatorname{softmax}(\alpha QK^{\mathsf{T}}) \tag*{(1)}

where the softmax is taken row-wise and α\alpha is a scalar. Since the head dimension dd is usually small, materializing the full attention matrix PP in global memory is a bottleneck. Therefore, fused implementations, like Flash Attention (FA) [2], never materialize PP in memory: they tile the sequence axis and accumulate the output in a single pass, so the two matrix multiplications QKTQK^{\mathsf{T}} and PVPV dominate the runtime of the layer. In this work we quantize these matrix multiplications.

In a tabular foundation model the primary attention bottleneck is the in-context learning (ICL) stage [13], which attends across rows of the dataset. The ICL stage is not a single attention call [5]. Writing NN for the number of training rows and MM for the number of test rows, keys and values are projected from the training rows alone, while queries are projected from all rows. The layer therefore evaluates two independent instances of (1): a train–train attention in which the NN training queries attend to the NN training keys, and a test–train attention in which the MM test queries attend to the same NN training keys. Overall, since the embedding dimension of the ICL stage is independent of features, the overall FLOPs of the model forward pass grow as O(N2+MN)\mathcal{O}(N^2 + MN). In contrast, the FLOPs associated with the weight matrix multiplications only grow linearly O(N+M)\mathcal{O}(N + M), so FP8 attention targets the dominant bottleneck that determines runtime for large NN, see Figure 3.

Figure 3. TabArena performance across three random seeds when Gaussian noise is added to QQ, KK, VV (left) and E2E runtime scaling.

We quantize to the e4m3fn FP8 format [10], whose range is [−448,448][-448,448]. To stay within the representable range, each quantized tensor XX is represented as X^≈sXX8\hat{X} \approx s_X X_8 with X8X_8 in FP8 and an FP32 dequantization scale sXs_X. We compute the scale dynamically from the tensor itself rather than using a static calibration. For each head hh we take the absolute maximum,

sX(h)=1448max⁡i,j∣Xij(h)∣,X8=rn⁡fp8(clamp⁡(X/sX,−448,448))(2)s_X^{(h)} = \frac{1}{448} \max_{i,j} \lvert X_{ij}^{(h)} \rvert, \qquad X_8 = \operatorname{rn}_{\mathrm{fp8}}\left(\operatorname{clamp}\left(X/s_X,-448,448\right)\right) \tag*{(2)}

where rn⁡fp8\operatorname{rn}_{\mathrm{fp8}} rounds to the nearest representable value. By construction the largest entry of each head maps to ±448\pm448, so the range is used in full. PP is the exception as it is not materialized outside of the kernel. However, since it is the softmax output, its range is known to be in (0,1](0,1], which wastes the exponent range above 1 and pushes the small probabilities towards subnormals. To circumvent this, we add a global constant to the exponent before exponentiation, which lifts the row maximum to 28=2562^8 = 256, below the 448 ceiling, as is also done in FA3 [?].

For the test–train attention, KK and VV are from the training rows and thus keep their scales. However, computing a dynamic scale on test queries couples the computation of the test rows. Although this does not measurably affect accuracy, it makes predictions depend on the exact test batch. We thus propose to use the scale computed on the train queries and reuse this for the test queries. Our experiments show that this works equally well and makes the test predictions batch invariant.

The baseline for our kernel is FA2 [1] called via torch scaled dot product attention (SDPA). Note that FA3 or FA4 do not apply to RTX Pro 6000. Our kernel is a Triton kernel and very closely follows the overall design of FA2 with the quantization similar to FA3 [Shah et al., 2024], see Figure 2. While there exist other low-precision attention kernels for other architectures [Zhang et al., 2025b,a, Shah et al., 2024], we note that previous low-precision kernels for Hopper and Ampere architectures had to work around an (implicit) low-precision accumulator [Zhang et al., 2025a, Section 3.4], which costs performance, but is not required for Blackwell chips we are targeting. Figure 1 compares the kernel speed to the FP16 baseline. The quantization overhead is amortized for N>8192N > 8192, so our final proposal gates on that.

FP8 attention kernel and end-to-end speedup, with accuracy and benchmark wall-clock results

Figure 1. FP8 attention kernel and end-to-end speedup (left). The kernel is turned on when there are more than 8192 train rows. The accuracy impact (right) on TabArena [3] and BeyondArena [12] is within seed noise, and we observe large wall-clock time speedups for the entire evaluation suite on BeyondArena including data preparation and scoring.

Figure 2. Quantization approach illustrated for a single head. We quantize QQ, KK, VV with per-head scaled absmax quantization, the QQ and KK scales are then applied in the kernel together with the softmax scale, whereas the VV scale is applied in the epilogue. The PP matrix is never materialized in global memory and is quantized with a fixed scale inside the kernel.

Experiments

For our main experiments we use TabPFN-v3 [5] and defer results on TabICLv2 [13], which confirm our findings, to the appendix. While our final proposal gates on N>8192N > 8192, Section 3.1 first investigates different strategies for quantization without the gate to investigate the accuracy impact of quantization independently of its latency aspects. All our experiments are run on RTX Pro 6000 Blackwell edition. For context, models on the TabArena leaderboard are typically separated by more than 10 Elo points.

Quality impact on TabArena

We first test the quality impact of our quantization scheme and kernel on TabArena [3], where we run the full suite consisting of 51 datasets. We experiment with applying FP8 to train↔train only, to test↔train only, or both, where we also distinguish between using the scale from the train queries also for the test queries. In addition to the effect size, we also compute a pp-value for the null hypothesis that the model did not get worse. Following Kübler et al. [7] we assume that post-training quantization can only make the model worse and use a one-sided sign test. We use over three seeds of TabPFN-v3’s preprocessing.

The aggregated results are in Table 1 and the full results in Table 2. We find that quantizing both attention calls minimizes the degradation, and the drops are almost negligible in practice. That said, our strict statistical treatment reveals that this tiny Elo drop likely is a small but detectable regression when the kernel is applied without gate. Furthermore, sharing the query scale from the train rows with the test rows does not alter the results in a relevant way, but makes predictions batch invariant, which is the approach we thus choose for the final experiments.

FP8 applied toΔ\Delta error (%)Δ\Delta Elosign-test pp
train↔\leftrightarrowtrain only+2.81 ±\pm 0.73−31.8 ±\pm 4.1<0.001–0.012
test↔\leftrightarrowtrain only+2.38 ±\pm 0.77−20.3 ±\pm 2.4<0.001–0.005
both, per-call query scale+0.01 ±\pm 0.03−1.0 ±\pm 0.40.024–0.131
both, shared train query scale+0.02 ±\pm 0.03−1.3 ±\pm 0.30.024–0.080

Table 1. FP8 attention ablations on TabPFN-v3 with TabArena without N>8192N > 8192 gating.

seednnwinΔ\Deltaerr (%)Δ\DeltaElosign pp
TabArena, all splits (51 datasets)
train↔train only0510.22+2.599−29.8<0.001
1510.33+2.206−29.20.012
2510.22+3.623−36.5<0.001
test↔train only0510.29+2.085−21.20.002
1510.24+1.802−17.5<0.001
2510.31+3.254−22.10.005
both, per-call query scale0510.35+0.029−1.40.024
1510.41−0.024−1.10.131
2510.35+0.027−0.60.024
both, shared train query scale0510.37+0.034−1.60.046
1510.35−0.013−1.20.024
2510.39+0.027−1.10.080
both, shared scale, gated0180.33+0.009−0.10.119
1180.44+0.003−0.50.407
2180.28+0.020−0.30.048
BeyondArena core (142 datasets)
both, shared scale, gated0560.46+0.004−2.00.344
1560.52−0.010+0.60.656
2560.59−0.011−0.10.930

Table 2. Accuracy ablations on TabPFN-v3. We run all the experiments over three seeds in TabPFN, whereas the datasets are not random. nn is the number of datasets whose predictions differ from the baseline at all: the gated arm leaves those below 8192 training rows untouched and hence they reproduce the baseline exactly. They are not considered in the sign test. win is the fraction of those nn on which the arm beat the baseline, so 0.5 is the null hypothesis. This is the quantity the sign test operates on. Δ\Deltaerr is the mean per-dataset relative change in metric_error, positive being worse. pp is the one-sided sign test pp-value.

Coordination between self-attention and cross-attention is needed

In Table 1 we find that it is critical to apply the same perturbation to both train and test rows. This might be counterintuitive, as we apply more local perturbation than if we just quantize one call. This is a remarkable feature of the architecture and to investigate whether it holds beyond quantization we conduct further experiments. We add random Gaussian noise to queries, keys, and values of the ICL layer, both in the individual calls only as well as on both calls. When applying it to both calls, we ablate independent noise as well as applying the same noise to KK and VV across attention calls (coordinated), see Section A.1. We then evaluate over TabArena with 3 different random seeds. The results are shown in Figure 3, and demonstrate that this is a model’s more general ability to sustain errors on the attention inputs as long as both attention calls receive the same version of KK and VV.

Latency measurements and final configuration experiments

We investigate the kernel latency and the resulting end-to-end latency. Quantization amortizes around 8,192 training rows. The kernel speedups reach 1.72x and the end-to-end speedups up to 1.67x. Full results are shown in Table 3 and Figure 1 visualizes the relative speedups. Figure 3 further shows the quadratic runtime scaling and the large absolute gains of using FP8. For completeness we also ablate a 16-bit variant of our kernel to ensure the speedups result from FP8 computation and not just better kernel design. Results are in Table 3 and show that our 16-bit variant closely follows the FA2 baseline. To show the generality of our kernel, in Section A.4 we also investigate head dimension 128 and L4 GPUs, for which our kernel provides even larger speedups.

Table 3: Speed of the kernel and end-to-end on TabPFN-v3 on NVIDIA RTX PRO 6000 Blackwell Server Edition. Our baseline is always torch SDPA, which uses FlashAttention 2. But for comparison we also include a 16-bit version of our own kernel, which closely follows torch. Therefore, the speedups can be clearly attributed to usage of FP8 tensor cores. Kernel: a single ICL train↔\leftrightarrowtrain attention call (B=1B = 1, H=Hkv=8H = H_{kv} = 8, D=64D = 64, float16), where quantization of QQ, KK, and VV is included. End-to-end: predict of a 20-feature, 1024-test-row problem. FP8 is gated off below 8192 training rows in the end-to-end tests.

NNtorchours 16-bitours FP8FP8 speedupbaselineFP8speedup
1k0.0650.0490.2160.30×0.1830.1890.97×
2k0.0970.0690.2390.40×0.2820.2741.03×
4k0.1950.1770.2950.66×0.3220.3221.00×
8k0.5040.4710.4701.07×0.4950.5010.99×
16k1.7811.6011.2161.46×0.9660.8951.08×
32k5.9835.8023.9621.51×2.3611.9381.22×
64k24.63323.98114.8771.66×7.2725.3091.37×
128k98.07195.98358.4021.68×24.43016.3501.49×
256k391.466384.460229.3491.71×87.75654.4121.61×
512k1579.2151552.739919.7821.72×334.499200.3751.67×

Table 3.

For the final runs we activate the kernel only if there are more than 8,192 train rows (gate) and share the query scale computed from the train rows with the test queries, to make the test predictions batch invariant. With that configuration we run TabArena again and also BeyondArena [12], again using three TabPFN preprocessing seeds. BeyondArena has 142 datasets out of which 56 are above the 8,192 gate. In particular it also has datasets with 1M training rows. Thus on this benchmark, while being essentially on par with the baseline, we see a substantial wall clock speedup, see Figure 1 and Table 2, and Appendix A.2 for a per-dataset resolution of the errors, showing that FP8 errors are smaller than preprocessing seed variations.

Conclusion and Outlook

We introduced attention quantization to tabular foundation models and showed that for datasets containing hundreds of thousands training rows, it can speed up inference end-to-end by up to 1.7x without relevant quality degradation on TabArena and BeyondArena. While faster prediction time increases usability, quantization also cuts the cost by a similar factor. This opens a new direction for post-training optimization of tabular foundation models and we found that coordination between train and test rows is required, a concept that is not present in LLMs.

Further Experimental Results and Details

Details on the Gaussian noise experiment

In the TabPFN-v3 architecture, test rows attend to the first K,VK,V head in a grouped-query attention style. The critical insight we find is that the calls should not receive different versions of that K,VK,V-pair.

To analyze this further, in Figure 3 we replaced quantization by an explicit random perturbation, so that we can the type of perturbation as well as its size. For a tensor X∈RS×H×dX \in\mathbb{R}^{S \times H \times d} (H=8H = 8 heads) and a strength ε>0\varepsilon> 0 we replace XX by

X~(h)=X(h)+ερ(h)Z(h),ρ(h)=(1Sd∑i,j(Xij(h))2)1/2,Zij(h)∼iidN(0,1),(3)\widetilde{X}^{(h)} = X^{(h)} + \varepsilon\rho^{(h)} Z^{(h)}, \qquad\rho^{(h)} = \left(\frac{1}{Sd}\sum_{i,j}\left(X_{ij}^{(h)}\right)^2\right)^{1/2}, \qquad Z_{ij}^{(h)} \mathrel{\overset{\mathrm{iid}}{\sim}} \mathcal{N}(0,1), \tag*{(3)}

independently for every head hh. The scale is set per head by that head’s own RMS ρ(h)\rho^{(h)}, mirroring the per-head scale sX(h)s_X^{(h)} of (2). With this, heads whose entries differ by orders of magnitude are perturbed equally in relative terms. We perturb QQ, KK and VV individually, and every attention layer draws its own field from a seeded generator, so a run is reproducible from the seed alone.

We sweep ε∈{1,2,3,4,5}×0.0265\varepsilon\in\{1, 2, 3, 4, 5\} \times0.0265, where 0.0265 is the error we found to usually arise from quantization. We thus swipe starting from an error similar to the quantization error.

We then ablate the following configurations:

  • train–train only: Only the train–train call receives perturbed inputs, while the test–train call gets QQ, KK, VV exactly.

  • test–train only: Only the test–train call receives perturbed inputs, while the train–train call gets QQ, KK, VV exactly.

  • both, independent: Both calls receive perturbed inputs, but the test–train call independently perturbs its KK and VV. Therefore the two calls see two different noisy versions of the same training keys and values.

  • both, coordinated: Both calls receive perturbed inputs, and the perturbation of KK and VV is applied once, before the head slice. The test–train call then consumes exactly the K,VK,V entries the train–train call consumed, whilst the QQ is still perturbed independently.

Per-dataset behavior of the gated final configuration

Figure 4 shows where the aggregate numbers of Table 2 come from. We show one point per dataset and seed, for the final configuration (gate at N>8192N > 8192, shared train query scale) against the unquantized baseline at the same seed.

Figure 4. Per-dataset relative change in metric error of the gated FP8 configuration against the unquantized baseline, over three preprocessing seeds. Grey points are datasets below the N>8192N > 8192 gate, which run the unquantized path and match the baseline. Green points are those where FP8 is active. Each bar spans the three seeds.

Below the gate, FP8 is not activated and the results match exactly. Above it, FP8 is active on 18 of 51 TabArena datasets and 56 of 142 BeyondArena datasets. There is no trend with dataset size, so for example, large datasets do not degrade more, although their speedup is larger. The gate is applied per run rather than per dataset. Since some datasets have splits that differ in size, it can engage on some splits and not others. We plot those datasets at their median size, which is why a marker can be left of the threshold.

We report relative error because the Arenas mix roc_auc, log_loss and rmse, whose absolute errors differ by orders of magnitude. The cost is that tiny movements on datasets that are almost optimal are inflated. The largest case in TabArena is APSFailure at around 50k training rows, scored by roc_auc at a baseline error of 0.0038–0.0069, i.e. AUC 0.993–0.996, where a relative error of +2% on a single split is an absolute AUC change of 8⋅10−58 \cdot10^{-5}. Furthermore, if we compare the error induced by TabPFN preprocessing over seeds, it is around five times larger than the within-seed FP8 deviations.

We further note that for BeyondArena, although there is a slight mean Elo drop, the FP8 model on average wins on the relative error. We do not claim though that the model got better, but attribute this to noise.

Experiments on TabICLv2

In the main body of the paper we did thorough ablations on TabPFNv3. To test whether FP8 attention works also on other models, and thus does not depend on specific architecture sizes, prior, and optimization choices, we also evaluate on TabICLv2 [13]. While TabPFNv3 has grouped query attention for the test rows and thus there naturally are two separate attention calls, TabICLv2 uses the same multi-head attention for both train and test rows. For simplicity we thus also use only a single attention call which we quantize with per-head dynamic scales.

For simplicity we also just use one implementation without gating on N>8192N > 8192 as this allows us to investigate the accuracy impact as a whole, which is interesting scientifically even if there is (currently) no speedup in this regime. There is one more subtlety which is that TabICLv2 runs very small datasets in FP32. To not confound our insights about FP8 quantization with moving from 32 bits to 16 bits, we do not enable the kernel for those datasets. We show the accuracy results in Table 4 and in Table 5 and they confirm our insights from TabPFN-v3. The attention quantization is nearly lossless, where in this case the pp-value is not significant. And the end to end speedup gains are also similar, reaching 1.56x on 512k train rows, which illustrates that on TabICLv2 other operations take a larger share of the overall latency than on TabPFNv3.

Table 4: FP8 ICL attention in TabICLv2 against stock TabICLv2. The baseline scores an Elo of 1568.9. To estimate the accuracy impact of quantization from 16 bit to FP8, we do not employ our previous gating of N>8192N > 8192. However, TabICLv2 already runs small datasets (N<1024N < 1024) in FP32. Since our goal is to understand the impact of quantization from 16 bits to FP8, and to not confound our insights with this design, we only replace the attention in the existing 16 bit paths. Columns are as in Table 2.

seednnwinΔ\Deltaerr (%)Δ\DeltaElosign pp
TabArena, all splits (51 datasets)
fp8, single ICL call (ungated)0460.43+0.009−1.80.231

Table 4.

Table 5: End to end speed of FP8 ICL attention in TabICLv2 on one NVIDIA RTX PRO 6000 Blackwell Server Edition. Same setting as in Table 3, however, for TabICLv2 we did not implement the actual gate at row count of 8k.

NNbaseline (s)FP8 (s)speedup
1k0.0620.0700.89×
2k0.0930.0970.96×
4k0.1550.1560.99×
8k0.2960.2921.01×
16k0.6350.6041.05×
32k1.4921.3341.12×
64k4.1183.3121.24×
128k13.0719.5171.37×
256k45.34230.6181.48×
512k168.044107.7341.56×

Table 5.

Generalization to different head dimensions and hardware

In the main paper we focused on RTX Pro 6000 and on head dimension 64 as this is used by both TabPFN-v3 and TabICLv2. To show that our kernel speedups generalize across hardware platforms and architectures, we also test our kernel on NVIDIA L4 GPUs as well as on head dimension 128. The results for head dimension 128 are shown in Table 6 and show speedups up to 1.82x which is larger than for the 64 case. The explanation is that the larger the head dimension the more the matrix multiplications dominate and other operations like application of scales or exponentiation matter less.

NNtime (ms)FP8 speedup
torchours 16-bitours FP8vs torchvs 16-bit
1k0.0860.0640.2490.34×\times0.26×\times
2k0.1460.1480.2850.51×\times0.52×\times
4k0.3690.3670.4380.84×\times0.84×\times
8k0.9900.9530.8471.17×\times1.13×\times
16k3.5443.4262.3371.52×\times1.47×\times
32k12.64512.4267.7471.63×\times1.60×\times
64k50.58850.06329.7921.70×\times1.68×\times
128k201.415200.501114.3951.76×\times1.75×\times
256k811.438811.546450.3501.80×\times1.80×\times
512k3273.2653279.9691800.4391.82×\times1.82×\times

Table 6. A single attention call at head dimension 128, on one NVIDIA RTX PRO 6000 Blackwell Server Edition.

For the L4 benchmarks on head dim 64 in Table 7 we even observe speedups over torch’s SDPA of up to 1.89x, though some of this can be attributed to its kernel not being perfectly optimized, as our own 16-bit variant is faster.

time (ms)FP8 speedup
NNtorchours 16-bitours FP8vs torchvs 16-bit
1k0.2510.1650.8590.29×0.19×
2k0.3230.2660.9210.35×0.29×
4k0.6760.6181.1440.59×0.54×
8k2.2732.2202.0511.11×1.08×
16k9.6598.6086.6681.45×1.29×
32k38.24135.35424.7841.54×1.43×
64k151.911138.45393.7031.62×1.48×
128k604.884555.437363.9731.66×1.53×
256k2613.4552300.4991443.5171.81×1.59×
512k11799.52310079.2516254.6351.89×1.61×

Table 7. A single attention call of our Triton kernel at head dimension 64, on one NVIDIA L4. torch is the SDPA baseline path (FlashAttention-2).

This brings us to limitations of our current kernel. For hardware like B200 or H100 the kernel would need to use special hardware units like the Tensor Memory Accelerator (TMA) to even reach 16 bit performance [?]. Since our kernel is an FA2 design, it does not use these. Furthermore, on B200 GPUs the tensor core performance has been scaled so drastically that even in 16 bit compute, the softmax can become the bottleneck [?]. In this case simply speeding up the matrix multiplications through FP8 would not result in meaningful speedups at all.

References

  1. [1]Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08691, 2023.arxiv.org/abs/2307.08691
  2. [2]Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 2022.
  3. [3]Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. T abarena: A living benchmark for machine learning on tabular data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. URL https://openreview.net/forum?id=jZqCqpCLdU.
  4. [4]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ICLR, 2023.arxiv.org/abs/2210.17323
  5. [5]Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Mihir Manium, Shi Bin Hoo, Magnus Bühler, Anurag Garg, et al. Tabpfn-3: Technical report. arXiv preprint arXiv:2605.13986, 2026.arxiv.org/abs/2605.13986
  6. [6]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025.DOI
  7. [7]Jonas M. Kübler, Kailash Budhathoki, Matthäus Kleindessner, Xiong Zhou, Junming Yin, Ashish Khetan, and George Karypis. When LLMs get significantly worse: A statistical approach to detect model degradations. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=cM3gsqEI4K.arxiv.org/abs/2602.10144
  8. [8]Eldar Kurtic, Alexandre Noll Marques, Shubhra Pandit, Mark Kurtz, and Dan Alistarh. “give me BF16 or give me death”? accuracy-performance trade-offs in LLM quantization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Vienna, Austria, July 2025. Association for Computational Linguistics.
  9. [9]Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon, Anna Leontjeva, and Simon Lucey. Memory efficient tabular foundation models, 2026. URL https://arxiv.org/abs/2607.27546.
  10. [10]Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv:2209.05433, 2022.arxiv.org/abs/2209.05433
  11. [11]Youngsuk Park, Kailash Budhathoki, Liangfu Chen, Jonas M. Kübler, Jiaji Huang, Matthäus Kleindessner, Jun Huan, Volkan Cevher, Yida Wang, and George Karypis. Inference optimization of foundation models on ai accelerators. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024.
  12. [12]Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzmüller, Alan Arazi, Alexander Pfefferle, Mustafa Tajpar, Gaël Varoquaux, Frank Hutter, et al. Beyond id: How general are tabular foundation models, really? arXiv preprint arXiv:2606.30410, 2026.
  13. [13]Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. In International Conference on Machine Learning, 2025.arxiv.org/abs/2502.05564

Paper details

vlnc:W_z3vf9nz4
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents