betaFeedback

WorksComputer Science

CARTA: Self-Supervised Reusable Transaction Embeddings via Heterogeneous Tokenization and Cross-Attention

Abstract

Acquiring fraud labels for payment transactions is expensive and inconsistent across tasks, and high-cardinality identifiers such as merchant names make one-hot feature pipelines infeasible at scale. We present CARTA, a self-supervised encoder that learns a compact 16-dim embedding from raw transactions without labels; the frozen embedding is then reused for label-efficient fraud probing and exploratory unsupervised risk stratification without retraining the encoder. CARTA uses type-specific projections with learned embedding tables for high-cardinality identifiers, trained with token excision and InfoNCE contrastive loss. On IEEE-CIS, its linear probe at 1% of labels exceeds a raw-feature probe at 100%; on IBM TabFormer (2M rows), its frozen embedding reaches GBDT AUPRC 0.508, above frequency encoding (0.411) and the merchant-dropping baseline (0.022), though fully-supervised CatBoost is stronger. CARTA is a label-free, reusable representation that complements task-specific supervised models.

CCS Concepts • Computing methodologies → Machine learning; • Information systems → Data mining.

Introduction

Banks and payment networks authorize billions of card transactions, and flagging the fraudulent ones is a core analytics task. The signal that separates fraud from legitimate spending seldom lies in any single field. For example, a $500 purchase is unremarkable on its own, but the same amount at an atypical merchant, at an odd hour, from a rarely-seen device can be highly suspicious. Useful models must therefore capture interactions among heterogeneous fields, often several at once.

Each authorization record is correspondingly high-dimensional and heterogeneous, spanning continuous monetary amounts (transaction value, settlement amount), low-cardinality flags (card type, payment method), timestamps (hour of day, day of week), and high-cardinality identifiers such as merchant name, city, and postal code. Encoding the categorical fields by one-hot encoding (OHE) inflates a single record to several hundred dimensions, and for the high-cardinality fields, with thousands of distinct values each, OHE becomes infeasible outright: one-hot encoding the merchant columns of a 2M-row benchmark yields a ~6,600-dimensional matrix that exceeds available memory.

Acquiring labeled fraud examples is expensive, delayed, and inconsistent across analytical tasks. Fraud labels typically arrive days after the transaction, rule-based labeling introduces noise, and the same transaction record that drives fraud scoring must also feed behavioral drift monitors and forecasting systems, each requiring its own bespoke feature-engineering pipeline. This per-task engineering must be reworked when new feature columns are added or removed, and creates inconsistent representations across the analytical stack.

We propose CARTA (Cross-Attention Reusable Transaction Autoencoder), a self-supervised system that learns once from raw transaction data and produces a single 16-dimensional embedding reusable across downstream tasks without retraining. It routes each feature type through an appropriate projection: continuous features through a learned dense layer, low-cardinality categoricals through a one-hot projection, and high-cardinality identifiers through learned embedding tables. The encoder is a Perceiver-style architecture [12] whose fixed set of latent query vectors cross-attend to the feature tokens, decoupling computational cost from input dimensionality; GRU-gated and MLP variants are evaluated as alternatives (§ 3.2).

The training methodology combines three elements: (i) Token excision, a self-supervised corruption augmentation that masks a random subset of feature tokens per view, so the encoder must rely on the surviving context (on IEEE-CIS this preforms on par with SCARF-style corruption); (ii) an InfoNCE contrastive loss [23] on differently-excised views of the same transaction, making the latent space discriminative; and (iii) cyclical KL annealing to prevent the posterior collapse that afflicts vanilla VAEs [15].

Contributions. Each component (Perceiver encoders, InfoNCE, VAE training, learned embeddings for high-cardinality categoricals) has precedent in isolation; their co-design for payment-transaction

This work is licensed under a Creative Commons Attribution 4.0 International License.

ICAIF ’26, Milan, AA, Italy

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-3087-0/2026/11

https://doi.org/10.1145/3858063.3858708 data is what yields properties unavailable from any single prior method. Concretely:

  1. Heterogeneous tokenization for finance: a type-specific projection layer (dense for continuous, OHE for low-cardinality, learned embedding tables [8] for high-cardinality identifiers) that avoids one-hot explosion without discarding merchant-identity signal.

  1. A single reusable, label-free embedding: one frozen 16-dim per-record representation evaluated across linear and GBDT probes, label-efficiency, and unsupervised clustering, benchmarked against PCA, UMAP, autoencoders and SCARF (Table 4), with full-label linear-probe quality similar to SCARF’s at a shorter single-run training time.

RepresentationLin. AUPRCLE@1%GBDT AUPRCSilhou.Train (s)
PCA-160.1120.1050.4550.152< 1
UMAP-160.0500.0460.3570.2021,318
Autoencoder0.1790.1470.4680.215319
Sparse AE0.1390.1350.4460.225303
MLP-VAE0.0830.0830.4430.717286
SCARF0.2140.2170.4910.086788
CARTA-16 (ours)0.2120.1870.4500.081506

Table 4. Unsupervised representation comparison on IEEE-CIS. Every method produces a 16-dim label-free embedding scored with the same probes, K-means silhouette (best k∈[3,7]k \in[3, 7]), and training wall-time. Lin. AUPRC is the mean over 5 probe seeds; LE@1% is that probe at 1% of training labels; each embedding is a single timed training run (UMAP timed multi-threaded). In this single-run comparison CARTA’s full-label linear AUPRC (0.212) is similar to SCARF’s (0.214) with a shorter wall-clock time (64% of SCARF’s), while SCARF leads on LE@1% (0.217 vs. 0.187) and GBDT AUPRC (0.491 vs. 0.450); the high-silhouette methods (MLP-VAE, UMAP) have weak probes, so silhouette does not track downstream usefulness here.

  1. Label-efficiency evidence: on a linear probe, the standard self-supervised measure of representation quality, CARTA at 1% of labels exceeds raw features at 100% of labels; gradient-boosted decision trees (GBDT) on raw features (Table 1) give a strong fully-supervised baseline for reference.

MethodDimLin. AUPRCGBDT AUPRCGBDT AUCLin. AUPRCGBDT AUPRCGBDT AUC
Raw†30/680.134±0.0020.616±0.0060.9230.013±0.0010.022±0.0020.835
Freq. enc.‡35/69———0.023±0.0010.411±0.0560.871
PCA-16160.112±0.0010.455±0.0050.8820.012±0.0000.017±0.0010.818
CARTA-16 (ours)160.217±0.0340.465±0.0130.8680.008±0.0090.508±0.0630.876
CatBoost§all—0.5080.907—0.7110.982

Table 1. Full-label probe performance (mean ± std); Dim = representation dimensionality. Bold marks the best frozen representation per column; the fully-supervised CatBoost baseline (§) is excluded. † Raw drops the high-cardinality columns (OHE infeasible); ‡ frequency encoding applies only to TabFormer. Subscript stds: CARTA-16 on IEEE-CIS is the mean over 5 independent training seeds (population std; matched inner probe scores averaged per run); Raw/PCA representations are deterministic, so their stds reflect 5 probe seeds. TabFormer results are from a single training run (5 probe seeds).

  1. An ablation study that evaluates the InfoNCE objective (removed jointly with its paired-view augmentation), the token-excision mode, and cyclical KL annealing, and compares three encoder architectures (Perceiver, GRU-gated, MLP), including cases where simpler alternatives are competitive.

Related Work

Sequential transaction transformers. TabBERT and TabFormer [18] apply masked-language-model pretraining over sequences of transactions, using a hierarchical transformer to capture both intrarecord field interactions and inter-record temporal patterns. FATA-Trans [25] extends this with field- and time-aware attention. These methods produce per-position sequence representations and are evaluated on a single downstream head; they do not produce a compact per-record embedding reusable across analytical tasks, and their O(N2)O(N^2) self-attention scales quadratically with sequence length. CARTA produces a fixed 16-dim per-record embedding in O(Q⋅N)O(Q \cdot N) where Q=8Q = 8 latent queries are independent of input dimensionality.

Self-supervised tabular learning. VIME [24] predicts mask vectors from corrupted inputs. SubTab [22] splits features into overlapping subsets and combines reconstruction with contrastive loss. SCARF [2] randomly corrupts individual feature values and applies SimCLR-style contrastive learning. CARTA uses token excision in the same spirit as these corruption schemes; in a controlled 5-training-seed comparison on IEEE-CIS (§ 5.3) cell-, feature-, and SCARF-style corruption perform comparably, so we treat the corruption scheme as a standard SSL component rather than a distinguishing one. TabTransformer [10], the closest prior, pretrains contextual categorical embeddings via self-attention but yields no compact reusable embedding. What CARTA adds over all of these is the combination of learned high-cardinality embedding tables with a compact, reusable per-record latent for payment-transaction data.

Supervised tabular transformers. FT-Transformer [7] tokenizes all features (continuous and categorical) and applies standard transformer self-attention, achieving strong results on supervised benchmarks. SAINT [21] extends this with inter-sample attention, and TabNet [1] uses sequential attention for feature selection. These are supervised architectures that require full label access and retraining per task; they do not produce reusable, task-agnostic self-supervised embeddings. For fraud detection specifically, Krutikov et al. [16] show that tabular transformers can approach GBDT performance with sufficient data.

Tabular foundation models. Cross-table pretrained models target transfer across datasets: TabPFN [9] (in-context inference), XTab [26] (a shared cross-table transformer), and CARTE [14] (graph attention over cell strings for high-cardinality entities). These emphasize cross-table generalization; CARTA instead learns one frozen, reusable per-record embedding for a single transaction stream. CARTE is closest for high-cardinality strings but targets cross-table transfer, not a reusable per-stream representation.

Method

Figure 1 overviews the pipeline: type-specific token projections, token excision into two views, a Perceiver encoder with a 16-dim VAE latent, and a composite VAE + InfoNCE objective, after which the frozen latent is reused across tasks.

Heterogeneous Feature Tokenization

Let a transaction record contain ncn_c continuous features, non_o low-cardinality categorical features (tractable for one-hot encoding), and nen_e high-cardinality categorical features (vocabularies too large for practical one-hot encoding, i.e., hundreds to thousands of distinct values). CARTA converts each group to a uniform dd-dimensional token space:

tcont=Wcxc∈Rd,(one token)(1)\mathbf{t}^{\mathrm{cont}} = \mathbf{W}_c \mathbf{x}_c \in\mathbb{R}^{d}, \qquad\text{(one token)} \tag*{(1)}
tohe=WoOHE⁡(xo)∈Rd,(one token)(2)\mathbf{t}^{\mathrm{ohe}} = \mathbf{W}_o \operatorname{OHE}(\mathbf{x}_o) \in\mathbb{R}^{d}, \qquad\text{(one token)} \tag*{(2)}
temb(i)=Ei[xe(i)]∈Rd,i=1,…,ne.(one token each)(3)\mathbf{t}^{(i)}_{\mathrm{emb}} = \mathbf{E}_i [\mathbf{x}^{(i)}_e] \in\mathbb{R}^{d}, \qquad i = 1,\ldots,n_e. \qquad\text{(one token each)} \tag*{(3)}

The resulting token sequence X=[tcont,tohe,temb(1),…,temb(ne)]∈RN×d\mathbf{X} = [\mathbf{t}^{\mathrm{cont}}, \mathbf{t}^{\mathrm{ohe}}, \mathbf{t}^{(1)}_{\mathrm{emb}},\ldots,\mathbf{t}^{(n_e)}_{\mathrm{emb}}] \in\mathbb{R}^{N \times d} with N=2+neN = 2 + n_e is augmented with learnable positional embeddings. Prior to tokenization, continuous features with heavy-tailed power-law distributions (detected via log-likelihood ratio test [4]) are transformed with log⁡(x+1)\log(x + 1) followed by Box-Cox normalization.

Perceiver Encoder and Variational Bottleneck

The core encoder adapts the Perceiver architecture [12] to tabular data. A set of Q=8Q = 8 learnable latent query vectors Q∈RQ×d\mathbf{Q} \in\mathbb{R}^{Q \times d} cross-attend to the feature tokens at each of L=4L = 4 encoder layers:

Q←LayerNorm⁡(Q+Dropout⁡(MHA⁡(Q,X,X))),(4)\mathbf{Q} \leftarrow\operatorname{LayerNorm}\left(\mathbf{Q} + \operatorname{Dropout}\left(\operatorname{MHA}(\mathbf{Q}, \mathbf{X}, \mathbf{X})\right)\right), \tag*{(4)}

where MHA is multi-head dot-product attention with 4 heads. Self-attention among latent queries, followed by a two-layer feed-forward network with GELU activations, allows the queries to specialize. After the final layer, the QQ latent vectors are mean-pooled to a single context vector c∈Rd\mathbf{c} \in\mathbb{R}^{d}, which is projected to the variational posterior parameters (μ,log⁡σ2)∈Rz(\boldsymbol{\mu}, \log\boldsymbol{\sigma}^2) \in\mathbb{R}^{z} with z=16z = 16.

The cross-attention complexity is O(Q⋅N)O(Q \cdot N) rather than O(N2)O(N^2); since Q=8Q = 8 is fixed, cost is linear in the number of input features.

Figure 1. CARTA system overview. Raw transaction features are projected into a uniform token space via type-specific transformations (continuous features to a dense projection, low-cardinality categoricals to a one-hot projection, and high-cardinality identifiers to learned embedding tables). Token excision randomly masks a subset of feature tokens per view as a self-supervised corruption. The Perceiver encoder compresses tokens into a 16-dim Variational Autoencoder (VAE) [15] latent. Training uses a composite loss (LVAE+LInfoNCE\mathcal{L}_{\mathrm{VAE}} + \mathcal{L}_{\mathrm{InfoNCE}} with cyclical KL). At inference the frozen embedding is reused across downstream tasks.

GRU-gated variant. An alternative encoder replaces the QQ latent queries with a single context vector c∈Rd\mathbf{c} \in\mathbb{R}^{d} iteratively refined across LL layers: at each layer the context cross-attends to the feature tokens, and the attention output is integrated via a GRU cell [3] with learned update and reset gates. This variant is more memory-efficient and is compared empirically in § 5.3.

Token Excision and Hybrid Training Objective

Token excision. During training, two corrupted views are created per transaction (p=0.2p=0.2). Cell mode independently zeroes individual continuous values and individual one-hot elements (element-wise Bernoulli masking before tokenization), leaving embedding indices unmasked (zeroing an integer index would map to a different category). Feature mode (default) masks whole feature groups in the input, also before tokenization: the continuous group is zeroed as a unit, each one-hot field’s block is zeroed as a unit, and embedding indices are replaced by index 0 (⟨unknown⟩), so corruption reaches high-cardinality embedding features. § 5.3 confirms all three modes are comparable on IEEE-CIS; feature mode is default because it is the only variant that masks embedding features.

Composite loss. The two excised views v1v_1, v2v_2 are independently encoded to latent means z1z_1, z2z_2. The total loss is:

L=12[LVAE(v1)+LVAE(v2)]+λcLInfoNCE(z1,z2),(5)\mathcal{L} = \frac{1}{2}\left[\mathcal{L}_{\mathrm{VAE}}(v_1) + \mathcal{L}_{\mathrm{VAE}}(v_2)\right] + \lambda_c \mathcal{L}_{\mathrm{InfoNCE}}(z_1,z_2), \tag*{(5)}

where LVAE=Lrecon+βDKL(q(z∣x)∥p(z))\mathcal{L}_{\mathrm{VAE}} = \mathcal{L}_{\mathrm{recon}} + \beta D_{\mathrm{KL}}(q(z|x) \mathbin{\|} p(z)) and the reconstruction loss is type-appropriate: L2 for continuous features; a selected-index softmax cross-entropy over the concatenated low-cardinality one-hot block, whose single integer target is the argmax of the concatenation — so only that selected index is directly supervised per record, not every low-cardinality field (a limitation of the implementation used for all reported results, under which the remaining low-cardinality fields are reconstructed only indirectly); and a separate per-vocabulary softmax cross-entropy for each high-cardinality embedding field. The InfoNCE loss treats the two views of the same transaction as a positive pair and all other transactions in the batch as negatives (temperature τ=0.07\tau=0.07, weight λc=0.1\lambda_c=0.1).

Cyclical KL annealing. To prevent posterior collapse, β\beta follows a cyclical schedule over C=10C=10 cycles, linearly increasing from 0 to βmax⁡=1\beta_{\max}=1 during the first half of each cycle. This allows the encoder to periodically escape collapse by temporarily reducing the KL penalty [6].

Decoder and Downstream Use

An MLP decoder maps zz back to the original feature space via type-specific output heads: L2 loss for continuous features; a shared logits head over the concatenated one-hot block trained with the selected-index cross-entropy of Section 3.3; and separate per-vocabulary heads for the high-cardinality features, each with its own cross-entropy. After training, only the encoder is deployed; the decoder is discarded. Downstream tasks consume the frozen 16-dim embedding vector μ\boldsymbol{\mu} (the posterior mean of the VAE), requiring no retraining.

Experimental Setup

Datasets

IEEE-CIS Fraud [11] (590,540 rows, 3.5% fraud): a widely used public card-transaction fraud benchmark. From its 400+ largely anonymized columns we retain 15 numeric and 3 low-cardinality categorical fields with low missingness (the full list is in the accompanying code); this retained subset has no high-cardinality identifiers, so the GBDT number here is an upper bound over this feature subset, not over all 400+ raw columns. Labels are known throughout but withheld from the encoder during pretraining.

TabFormer (2,000,000 rows sampled, 0.13% fraud): the synthetic credit-card transaction dataset released with TabBERT [18]. The original release includes two datasets; we use the credit-card subset, referred to as “TabFormer” throughout. It contains 1 continuous feature (transaction amount), 5 low-cardinality categoricals (use chip, error type, hour, weekday, month), and 5 high-cardinality identifiers (Merchant Category Code (MCC) 109 values, merchant state 200 values, merchant name 1,397 values, merchant city 2,075 values, ZIP code 3,109 values); one-hot encoding these columns is infeasible at 2M rows (the ∼6,600-dimensional matrix exceeds available memory). Categories appearing fewer than 100 times are collapsed to an other sentinel, following standard practice.

Baselines

  • Raw: z-score standardization of numeric features plus one-hot encoding of low-cardinality categoricals (high-cardinality columns excluded, as dense one-hot encoding is infeasible at this scale). This is a common practitioner default; other categorical encoders (target or hashing encoders) are not evaluated here.

  • PCA-16: Principal Component Analysis (PCA) applied to the raw matrix, reducing to 16 dimensions, the classical linear dimensionality-reduction baseline at the same latent size as CARTA.

  • CatBoost [20]: a gradient-boosted decision tree with native high-cardinality categorical handling (ordered target statistics), trained fully supervised on all features, including the merchant identifiers that Raw must drop. Unlike CARTA it requires task labels and is retrained per task; it is our strongest fully-supervised baseline (200 iterations, depth 6, balanced class weights, 5 seeds, identical splits).

  • CARTA-16: frozen 16-dim latent means from the trained CARTA encoder.

GBDT denotes gradient-boosted decision trees [5], using the histogram-based scikit-learn [19] implementation (max depth 6, 200 estimators). Results are means ± standard deviations with stratified 70/30 train/test splits; each table states whether the reported variation is over training seeds (independent encoder runs) or probe seeds (re-fit probes on a fixed representation).

Evaluation Protocol

All representations (raw, PCA-16, and CARTA-16) are evaluated through the same two probes: (i) a linear probe (logistic regression, class-weight balanced), measuring pure representation quality; and (ii) a GBDT probe, measuring the upper-bound downstream utility. All representations are evaluated with frozen parameters; no labels are used during the representation learning phase of CARTA or PCA.

Our primary metric is the area under the precision–recall curve (AUPRC; equivalently average precision). We also report the area under the ROC curve (ROC-AUC) for the GBDT probe. AUPRC is the appropriate metric for highly imbalanced fraud detection, since ROC-AUC can be misleadingly high even for weak classifiers [13].

Why linear probes. Following standard SSL practice [23], the linear probe measures representation quality: a frozen embedding is good when a simple linear classifier separates the classes from it. GBDT on raw features (Table 1) is reported alongside as a strong fully-supervised baseline.

Label-efficiency evaluation. To assess the practical value of self-supervised pretraining, each probe is also trained on stratified subsets of 1%, 5%, 10%, 25%, and 100% of training labels, with the test set held fixed.

Implementation. All CARTA models are trained on a single NVIDIA A100 GPU using JAX/Flax, batch size 16,384, Adam optimizer with warmup-cosine schedule (peak 5×10−45 \times10^{-4}, 10 warmup epochs), up to 300 epochs with early stopping (patience 50) on held-out self-supervised loss. Hyperparameters (Q=8Q=8, L=4L=4, z=16z=16, p=0.2p=0.2, τ=0.07\tau=0.07, λc=0.1\lambda_c=0.1) are standard defaults held fixed across datasets. Code and configuration files are available at https://github.com/Shreyaskc/carta; both benchmarks are publicly available.

Results

Main Representation-Quality Results

Table 1 reports full-label probe performance.

IEEE-CIS. On the linear probe, CARTA-16 (0.217) outperforms raw features (0.134) despite using half the dimensions: self-supervised pretraining yields a more linearly separable representation than the raw feature space. On the GBDT probe, raw features (0.616) retain an advantage, as expected for a supervised GBDT with full feature access; CARTA’s value lies in label efficiency and multi-task reuse (§ 5.2).

TabFormer. The high-cardinality experiment illustrates two points. First, the practical raw baseline, which must exclude merchant name (1,397 unique values), city (2,075), and ZIP (3,109) because one-hot encoding them at 2M rows exceeds available memory, achieves only GBDT AUPRC = 0.022, far short of the encodings that retain merchant identity. CARTA-16, encoding all merchant identifiers via learned embedding tables, achieves GBDT AUPRC = 0.508, a factor of 23 improvement over the raw baseline that excludes merchant features entirely. A common alternative to one-hot encoding replaces high-cardinality categories with log-normalized frequency encoding [7], which achieves GBDT AUPRC = 0.411 at full 2M-row scale (Table 1). CARTA (0.508) still outperforms this stronger baseline, indicating that learned embedding tables capture richer merchant-identity signal than frequency proxies. The merchant identifiers carry strong fraud signal (fraud incidence is concentrated among subsets of merchants); among the baselines evaluated here, CARTA is the only one that incorporates this signal via learned embedding tables without requiring full label supervision. The weak CARTA linear headline (0.008) reflects that at a 0.13% base rate a class-weighted linear head over a 16-dim latent recovers little; the merchant-identity signal in the embedding is nonlinear and is extracted by the GBDT probe. Note: the extreme class imbalance (0.13% fraud, ∼760 test-set positives) produces high variance; for merchant-attribute representations, GBDT AUPRC standard deviations across the 5 probe seeds (re-split and re-fit on a fixed embedding) range from 0.02 to 0.08.

Method1%5%10%25%100%
Raw0.1620.1290.1310.1360.134
PCA-160.1050.1110.1100.1140.112
UMAP-160.0460.0490.0490.0500.050
CARTA-160.2060.2190.2150.2190.217

Table 2. Linear-probe AUPRC on IEEE-CIS vs. fraction of training labels (mean; 5 training seeds for CARTA-16, 5 probe seeds otherwise). CARTA-16 at 1% labels exceeds raw features at 100% labels; the margin (CARTA-16 at 1% = 0.206 vs. raw at 100% = 0.134) exceeds the full-label stds (±0.034 over training seeds for CARTA-16 and ±0.002 over probe seeds for raw, respectively).

Figure 2. Linear-probe AUPRC vs. fraction of training labels; error bars are ±σ over 5 training seeds for CARTA-16 and 5 probe seeds for Raw and PCA-16. On IEEE-CIS, CARTA-16 at 1% of labels (0.206) already beats raw features at 100% of labels (0.134), so self-supervised pretraining reaches a given representation quality with far fewer labels.

Fully-supervised baseline (CatBoost). CatBoost, trained end-to-end on all features including native categoricals, is the strongest baseline on TabFormer (GBDT AUPRC = 0.711), exceeding CARTA’s frozen embedding (0.508). CatBoost consumes task labels for this single task, while CARTA’s embedding is label-free and reused across tasks, so the two occupy different points on the supervision spectrum. On IEEE-CIS (all low-cardinality), CatBoost (0.508) sits below raw-feature GBDT (0.616), reflecting hyperparameter defaults (symmetric trees vs. depth-6 sklearn GBDT); this is consistent with prior reports [20]. The high-cardinality comparison is therefore three-way: both learned embedding tables (CARTA, 0.508) and native-categorical boosting (CatBoost, 0.711) far outperform the merchant-dropping raw baseline (0.022) and the frequency-encoding proxy (0.411). CatBoost attains the highest single-task accuracy, and among the four methods CARTA is the one that yields a label-free, reusable representation.

Label Efficiency

Figure 2 and Table 2 show linear-probe AUPRC as the fraction of training labels is varied.

On IEEE-CIS, CARTA at 1% of labels (AUPRC = 0.206) exceeds raw features at 100% (0.134 ± 0.002); at a matched 1% budget it also leads raw (0.162), so the advantage does not hinge on the cross-fraction comparison. Both curves are mildly non-monotonic: at a 3.5% base rate a class-weighted linear head has high variance at small support and sits near that base rate, so more labels need not monotonically help it. CARTA front-loads the discriminative structure so few labels suffice, while PCA-16 reaches only 0.105 at 1%; the gain therefore comes from the learned nonlinear compression, since a generic 16-dimensional projection does not achieve it.

Ablation Study

Table 3 compares training variants; the no-InfoNCE variant also removes paired-view exicsion, so its effect cannot be attributed to the contrastive loss alone.

InfoNCE contrastive objective. The no-InfoNCE variant shows the largest linear-probe drop on both datasets: on IEEE-CIS from 0.199 to 0.151 (−0.048 AUPRC); on TabFormer from 0.014 to 0.005 (about a 3× relative decrease). This variant trains on a single uncorrupted view, so the drop measures the combined effect of the contrastive loss and its paired-view excision augmentation, not the loss in isolation.

Cyclical KL annealing. Replacing the cyclical schedule with a fixed β=1\beta=1 (no annealing) slightly improves IEEE-CIS linear AUPRC (0.208 vs. 0.199) and markedly improves TabFormer (0.219 vs. 0.014). Even in the label-efficient regime the two are tied on IEEE-CIS (1%-label linear AUPRC 0.219 with annealing vs. 0.215 without). We therefore make no representation-quality claim for cyclical annealing: its documented role is to stabilize VAE training against posterior collapse [6], and we keep it in the Full configuration for collapse prevention and one consistent recipe across datasets, not for any downstream gain. The large TabFormer gap, and whether the no-annealing gains survive retuning, warrant a dedicated schedule study we leave to future work.

Token excision. The “− Token excision” row sets p=0p=0 while keeping InfoNCE; the two views then differ only via VAE reparameterisation and dropout, giving a weak contrastive signal (linear probe: 0.186 / 0.031).

Excision mode and SCARF: a controlled 5-seed comparison. We compare feature-wise and cell-mode token excision against SCARF-style marginal shuffling [2], under an identical Perceiver encoder and budget and each trained over 5 independent seeds (IEEE-CIS only; training-seed std). On IEEE-CIS the three linear-probe means are feature-wise 0.217±0.034, cell-mode 0.227±0.039, and SCARF 0.197±0.022: a spread comparable to the per-mode training-seed standard deviations, so with 5 seeds per mode we observe no clear separation and conclude the excision granularity does not drive quality on this low-embedding dataset. The choice does matter on embedding-heavy data. Cell-mode never masks the learned embedding tokens, so on TabFormer it corrupts only the continuous and low-cardinality tokens, whereas feature-wise excision also masks whole embedding features and attains the strongest TabFormer high-cardinality GBDT (Table 1). We therefore adopt feature-wise excision as the default, not for a quality edge on IEEE, but because it is what makes token excision act on high-cardinality embedding features. An earlier single-seed reading had suggested a broader feature-wise advantage on IEEE; that did not survive multi-seed evaluation, which we report for transparency.

Encoder architecture. All three encoder architectures (Perceiver, GRU-gated, MLP-VAE) are competitive. The GRU encoder leads on TabFormer GBDT AUPRC (0.510, vs. MLP 0.487 and Perceiver 0.401) and on IEEE-CIS linear AUPRC (0.212, vs. MLP 0.203 and Perceiver 0.199). At N=7N=7 tokens, the GRU’s iterative context refinement over a short token sequence is as effective as maintaining Q=8Q=8 parallel latent queries; the MLP is also competitive on IEEE-CIS GBDT (0.499). All three architectures share the same tokenization, token excision, and training objective, so the differences reflect encoder capacity rather than any other factor. The Perceiver’s O(QN)O(QN) decoupled complexity becomes advantageous only as feature count grows toward the 30+ heterogeneous columns of full industrial card-network datasets; at small NN the simpler baselines are credible alternatives. We thus present the Perceiver as the most scalable of the three, competitive at small NN and increasingly advantageous as the feature-token count grows.

Comparison to Unsupervised Representation Baselines

Tables 1–3 compare CARTA against supervised models (raw-feature GBDT, CatBoost); a representation, though, is best judged against other representations. We therefore compare CARTA to seven label-free encoders that all produce a 16-dim embedding from the same inputs, scoring each with the same linear and GBDT probes, K-means silhouette, and training-time budget (Table 4, Figure 3): PCA and Uniform Manifold Approximation and Projection (UMAP) [17] (non-linear dimensionality reduction), a plain autoencoder (AE), a sparse autoencoder (SAE), a vanilla MLP-VAE (no excision or contrastive objective), and SCARF [2]. Supervised GBDT and CatBoost stay a separate upper reference (§ 6).

Figure 3. Representation quality (linear-probe AUPRC, 5 probe seeds) versus training wall-clock time for seven label-free 16-dim encoders on IEEE-CIS (one timed training run per method). CARTA’s (blue) full-label linear AUPRC is similar to SCARF’s at 64% of SCARF’s single-run training time. Error bars are ±σ\pm\sigma over probe seeds.

CARTA and SCARF lead on the full-label linear probe, and both encode non-linear structure. Among label-free encoders, CARTA and SCARF attain the highest linear-probe AUPRC (0.212 and 0.214), roughly 1.9× the score of a PCA projection (0.112) and well above the autoencoder (0.179), sparse autoencoder (0.139), and MLP-VAE (0.083). Non-linearity alone does not explain the gap: UMAP-16, a non-linear manifold embedding of the same inputs, is the weakest representation (0.050, below even linear PCA), because it optimizes local neighborhood structure for visualization rather than linear separability of a downstream signal. The advantage therefore comes from the self-supervised objective, not from non-linearity per se.

CARTA reached this full-label linear-probe quality with less compute in our single timed run: 506 s versus 788 s for SCARF (which took 56% longer), plausibly because token excision is cheaper than SCARF’s marginal shuffling, which resamples each corrupted cell from the empirical column distribution. These are one timing per method on one machine, not a general speed claim, and SCARF leads on LE@1% (0.217 vs. 0.187) and the GBDT probe (0.491 vs.

0.450) in the same comparison. The controlled multi-seed excision-vs-SCARF quality comparison is reported in § 5.3.

Silhouette does not track representation quality under extreme imbalance. The MLP-VAE attains by far the highest silhouette (0.717) yet a weak linear probe (0.083), and UMAP-16 pairs a moderate silhouette (0.202) with the lowest probe of all (0.050), while CARTA and SCARF have the lowest silhouettes (0.081, 0.086) yet the strongest probes. Cluster compactness is thus a poor proxy for fraud-relevant structure at a 3.5% base rate, so we judge representation quality with supervised probes rather than a projection visual.

Latent Space Structure and Unsupervised Risk Stratification

Figure 4 shows unsupervised risk stratification on IEEE-CIS: K-means on the 16-dim latent (no fraud labels) recovers 3 behavioral archetypes with a 4.0× fraud-rate spread, and the clustering is stable (pairwise ARI = 0.9991 ± 0.0005 over 10 seeds). As in § 5.4, the low silhouette (0.085) reflects archetype overlap that nonetheless differs sharply in fraud rate; the spread carries the discriminative signal that could support analyst triage.

Figure 4. Unsupervised behavioral archetypes discovered by K-means (k=3k = 3, silhouette = 0.085) on the full IEEE-CIS CARTA latent space (590K transactions, no fraud labels used). The highest-incidence archetype (103K transactions) shows an observed fraud rate of 8.4%, nearly 4.0× the rate of the lowest-incidence archetype (2.1%), despite being discovered without any labeled supervision.

The same clustering on the 2M-transaction TabFormer latent (k=4k = 4, silhouette = 0.141) recovers merchant-type archetypes with a factor of more than 20 variation in observed fraud incidence across groups (from 0.01% to 0.21%), against a 0.127% overall rate. Since merchant identity is itself an encoder input, recovering merchant-type groupings is expected by construction; the fraud-incidence spread across groups is not directly supervised and may partly reflect transaction frequency or merchant mix, so we treat these clusters as exploratory groupings rather than causal fraud signals.

Discussion and Limitations

Fully-supervised gradient boosting is stronger on peak accuracy. On IEEE-CIS, raw-feature GBDT (0.616) leads; on TabFormer, CatBoost with native categoricals (0.711) is strongest. CARTA does not beat these fully-supervised, single-task models on peak accuracy and we do not claim it does (CatBoost is itself label-efficient, IEEE 1%-label AUPRC 0.348). Its contribution is complementary: a single label-free embedding reused across the analytical stack with no per-task labels or retraining. With abundant labels for one fixed task, CatBoost is the better choice; when one consistent representation is needed across many evolving tasks, CARTA is the setting we target.

Class imbalance and AUPRC variance. TabFormer’s 0.13% fraud rate leaves ∼\sim760 test positives, so AUPRC is sensitive to seed noise (GBDT AUPRC std 0.02–0.08 over 5 probe seeds for the merchant-aware representations). A factor of 23 improvement over raw is robust, though per-variant TabFormer differences should be read as indicative: every TabFormer number comes from a single trained encoder per variant, and replication over independent training seeds was performed only on IEEE-CIS, so a larger sweep on this imbalanced benchmark is warranted.

Evaluation splits and potential leakage. Our primary splits are random 70/30. Because the original IEEE-CIS split is temporal, random splits are optimistic for time-ordered fraud; learned merchant embeddings also permit entity leakage when the same merchants appear in both train and test. All representations are fit on the pooled feature matrix without labels and each probe is trained only on the training split, so the transductive feature setting is uniform across methods and no labels enter representation learning. We also evaluate under a temporal split (last 30% by date as test): the linear-probe advantage persists (CARTA 0.240 vs. raw 0.180; the matched single-run random split gives 0.198/0.134), so the representation-quality finding is not a random-split artifact. Merchant-disjoint TabFormer splits remain as future work.

Few tokens on the evaluated datasets. Both benchmarks are small (N=2N=2 on IEEE-CIS, N=7N=7 on TabFormer), so the Perceiver’s O(Q⋅N)O(Q \cdot N) advantage is not exercised and an MLP encoder is competitive (Table 3). The architecture targets the 30+ heterogeneous-column regime of production data; validation there is future work.

VariantIEEE-CIS Lin. AUPRCIEEE-CIS GBDT AUPRCTabFormer Lin. AUPRCTabFormer GBDT AUPRC
Full (cell + InfoNCE + cycl. KL)0.1990.4920.0140.401
− Cyclical KL (β=1\beta=1)0.2080.4810.2190.431
− InfoNCE (single view)0.1510.4670.0050.365
− Token excision0.1860.4250.0310.513
Feature-wise excision (default)0.2410.4690.0090.533
GRU encoder0.2120.4940.0150.510
MLP encoder (no attention)0.2030.4990.0080.487

Table 3. Ablation results (linear probe / GBDT AUPRC; each variant is a single trained encoder, Perceiver unless noted, evaluated as the mean over 5 probe seeds; up to 300 epochs with early stopping, batch 16,384 throughout). “Full” denotes the ablation reference configuration (cell-mode excision + InfoNCE + cyclical KL). Rows are intended for within-table comparison; the canonical CARTA numbers are those in Table 1, whose CARTA row instead averages 5 independent training runs with per-run inner-probe averaging. The protocols therefore differ by seed population (e.g., the feature-wise row here is one training run scoring 0.241, while the Table 1 5-run mean is 0.217 ± 0.034). TabFormer values are especially sensitive: the linear probe operates near the 0.13% base rate, and for the merchant-aware variants GBDT AUPRC varies by 0.02–0.08 (std) across the 5 probe seeds even on a fixed embedding.

Cold start. Unseen merchant identifiers map to a reserved ⟨unknown⟩ embedding index at inference, distinct from the OTHER bucket that collapses rare categories during vocabulary fitting; newly appearing merchants can be the riskiest, so the embedding advantage is smallest where it could matter most. Hashing or side-information fallbacks are future work.

Scope and comparison breadth. We evaluate the frozen embedding on two downstream tasks (label-efficient fraud probing and unsupervised risk stratification); drift monitoring and forecasting remain motivating uses we do not yet evaluate. An end-to-end FT-Transformer [7] on IEEE-CIS reaches AUPRC 0.253 at 1% and 0.696 at 100% of labels but is supervised and per-task (no reusable embedding). Table 4 covers comparable SSL methods (SCARF, AE, SAE); a full comparison to VIME and SubTab is a natural follow-on. The latent width z=16z=16 was chosen to match PCA-16 and not tuned; a sensitivity study is straightforward with the released code. All results are reported on public research benchmarks and reflect research findings, not organizational policy or operational practice.

Conclusion

We presented CARTA, a self-supervised encoder for compact, reusable embeddings from raw heterogeneous payment transaction data. On two public benchmarks, CARTA achieves label efficiency (1% of labels exceeds a raw-feature probe at 100%) and high-cardinality capability (outperforming baselines that must drop merchant identifiers and frequency-encoding proxies that replace them), while fully-supervised CatBoost remains stronger on peak single-task accuracy. In the ablation, removing the contrastive objective together with its paired-view augmentation (a compound intervention) is the most damaging change; the spread across token-excision modes on IEEE-CIS is comparable to per-mode training-seed variability, with feature-wise excision retained to extend corruption to embedding features. Unsupervised K-means on the frozen latent recovers behavioral archetypes whose observed fraud incidence differs markedly across groups without label supervision. The encoder is trained once and reused across downstream

References

  1. [1]Sercan Ö. Arik and Tomas Pfister. 2021. TabNet: Attentive Interpretable Tabular Learning. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI). AAAI Press, 6679–6687. doi:10.1609/aaai.v35i8.16826DOI
  2. [2]Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. 2022. SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption. In Proceedings of the 10th International Conference on Learning Representations (ICLR). OpenReview.net.arxiv.org/abs/2106.15147
  3. [3]Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 1724–1734. doi:10.3115/v1/D14-1179DOI
  4. [4]Aaron Clauset, Cosma Rohilla Shalizi, and Mark E. J. Newman. 2009. Power-Law Distributions in Empirical Data. SIAM Rev. 51, 4 (2009), 661–703. doi:10.1137/070710111DOI
  5. [5]Jerome H. Friedman. 2001. Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics 29, 5 (2001), 1189–1232. doi:10.1214/aos/1013203451DOI
  6. [6]Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Association for Computational Linguistics, 240–250. doi:10.18653/v1/N19-1021DOI
  7. [7]Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting Deep Learning Models for Tabular Data. In Advances in Neural Information Processing Systems 34 (NeurIPS). Curran Associates, Inc., 18932–18943.arxiv.org/abs/2106.11959
  8. [8]Cheng Guo and Felix Berkhahn. 2016. Entity Embeddings of Categorical Variables. arXiv preprint arXiv:1604.06737 (2016).arxiv.org/abs/1604.06737
  9. [9]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. 2025. Accurate predictions on small data with a tabular foundation model. Nature 637 (2025), 319–326. doi:10.1038/s41586-024-08328-6
  10. [10]Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. 2020. TabTransformer: Tabular Data Modeling using Contextual Embeddings. arXiv preprint arXiv:2012.06678 (2020).arxiv.org/abs/2012.06678
  11. [11]IEEE Computational Intelligence Society and Vesta Corporation. 2019. IEEE-CIS Fraud Detection. Kaggle competition. https://www.kaggle.com/c/ieee-fraud-detection.
  12. [12]Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and João Carreira. 2021. Perceiver: General Reception with Iterative Attention. In Proceedings of the 38th International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 139). PMLR, 4651–4664.
  13. [13]Sérgio Jesus, José Pombal, Duarte Alves, André Cruz, Pedro Saleiro, Rita P. Ribeiro, João Gama, and Pedro Bizarro. 2022. Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation. In Advances in Neural Information Processing Systems 35 (NeurIPS), Datasets and Benchmarks Track. Curran Associates, Inc.
  14. [14]Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. 2024. CARTE: Pretraining and Transfer for Tabular Learning. In Proceedings of the 41st International Conference on Machine Learning (ICML).arxiv.org/abs/2402.16785
  15. [15]Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR).arxiv.org/abs/1312.6114
  16. [16]Sergei Krutikov, Bulat Khaertdinov, Rodion Kirikukhin, Shubham Agrawal, and Kees Jan de Vries. 2024. Challenging Gradient Boosted Decision Trees with Tabular Transformers for Fraud Detection at Booking.com. arXiv preprint arXiv:2405.13692 (2024).
  17. [17]Leland McInnes, John Healy, and James Melville. 2018. UMAP: Uniform Manifold Approximation and Projection. arXiv:1802.03426 (2018).arxiv.org/abs/1802.03426
  18. [18]Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. 2021. Tabular Transformers for Modeling Multivariate Time Series. In Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3565–3569. doi:10.1109/ICASSP39728.2021.9414142DOI
  19. [19]Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.arxiv.org/abs/1201.0490
  20. [20]Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2018. CatBoost: Unbiased Boosting with Categorical Features. In Advances in Neural Information Processing Systems 31 (NeurIPS). Curran Associates, Inc., 6639–6649.arxiv.org/abs/1706.09516
  21. [21]Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. 2021. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training. arXiv preprint arXiv:2106.01342 (2021).
  22. [22]Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. 2021. SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning. In Advances in Neural Information Processing Systems 34 (NeurIPS). Curran Associates, Inc., 18853–18865.arxiv.org/abs/2110.04361
  23. [23]Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748 (2018).arxiv.org/abs/1807.03748
  24. [24]Jinsung Yoon, Yao Zhang, James Jordan, and Mihaela van der Schaar. 2020. VIME: Extending the Success of Self- and Semi-Supervised Learning to Tabular Domain. In Advances in Neural Information Processing Systems 33 (NeurIPS). Curran Associates, Inc.
  25. [25]Dongyu Zhang, Liang Wang, Xin Dai, Shubham Jain, Junpeng Wang, Yujie Fan, Chin-Chia Michael Yeh, Yan Zheng, Zhongfang Zhuang, and Wei Zhang. 2023. FATA-Trans: Field and Time-Aware Transformer for Sequential Tabular Data. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM). ACM, 3247–3256. doi:10.1145/3583780.3614879DOI
  26. [26]Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karpius, and Mahsa Shoaran. 2023. XTab: Cross-table Pretraining for Tabular Transformers. In Proceedings of the 4th International Conference on Machine Learning (ICML).arxiv.org/abs/2305.06090

Paper details

vlnc:W_8z8wnz64
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents