betaFeedback

WorksComputer Science

Experts Across Schemas: Agent-Guided Mixture-of-Relational-Experts for Cross-Database Generalization

Abstract

Web platforms record user and item activity in relational databases whose schemas differ from platform to platform, which makes it difficult to transfer predictive models between them. Relational foundation models address this through cross-database pretraining, yet recent improvements have relied on increasingly large pretraining corpora. We ask whether a better architecture, together with features specific to the target database, can deliver comparable gains from far less pretraining data. We introduce MoREGT, a mixture-of-relational-experts graph transformer that represents both rows and cells as tokens, adds the log-ages of rows and the names of foreign-key relationships to attention, and routes cells and rows to experts by column metadata rather than by values. A label-blind LLM agent complements the model: it reads the target database's schema and column statistics and adds validated single-row derived columns, without access to labels or model scores. On 21 RelBench tasks with leave-one-database-out pretraining on six databases, the MoREGT backbone improves on the Relational Transformer pretrained on the same databases by 2.25 points of mean AUROC and 5.83 points of mean nMAE. The agent's columns raise mean AUROC from 78.45 to 79.63 and lower mean nMAE from 27.91 to 26.98, the highest means on both metrics among the compared methods, including RT-J, which pretrains on about 400 times as many databases.

Introduction

Web platforms such as online retailers, classified-ad marketplaces, question-answering forums and event services record user activity in relational databases ([22]). Many routine decisions on these platforms depend on predictions about users and items. An online retailer asks whether a customer will write no reviews over the next three months; a question-answering forum asks whether a user will post, comment or vote over the same period ([22]). The two databases share no columns: the retailer stores customers, products and reviews, while the forum stores users, posts, comments and votes. Yet both predictions rest on the same kind of evidence, a user’s history of timestamped rows linked through foreign keys to other rows (Figure 1). This poses a central challenge for Web graph representation learning: transferring predictive patterns over heterogeneous temporal structured data across platforms without requiring shared entities, relationships, or database schemas.

Figure 1. Two RelBench databases with no shared columns ask whether a user will stay active over the next three months. Left: online retailer rel-amazon (customers, products, reviews). Right: question-answering forum rel-stack (users, posts, comments, votes and other tables). In both, the evidence is the user’s history of timestamped rows linked by foreign keys.

Relational deep learning (RDL) learns such predictions directly from the database, treating rows as nodes of a temporal graph and foreign keys as edges, without hand-built features ((Fey et al., 2024). Recent RDL models, including graph neural networks [1] and relational graph transformers [4, 5], are usually trained for one schema and one task, so each new platform needs its own model. Relational foundation models such as Relational Transformer (RT), KumoRFM, and RT-J aim to reduce this cost by pretraining on many databases and predicting on unseen ones [20, 14, 21]. For Web platforms with diverse relational schemas, such transfer raises two challenges. The first is learning structure that schemas share: column names and semantics differ, but entities still have histories connected through relationships and time, and a review written last week usually says more about whether a customer will stay than one written three years ago (Figure 1). A model that transfers this structure must represent elapsed time between events, treat rows as units that linked rows can attend to, and accommodate heterogeneous column semantics without forcing every column through the same feed-forward parameters. The RT backbone [20] handles each indirectly. PluRel scales it with synthetic databases [19] and RT-J with real databases, a larger model and longer contexts [21], but both leave these representations unchanged. The second challenge is structure specific to the target database. Quantities such as code groups or rates between columns may not appear during pretraining and can be hard to learn from task labels alone [8, 15], and adding them by hand would bring back the manual feature engineering that RDL was designed to remove [7, 22].

To this end, we introduce MoREGT, a mixture-of-relational-experts graph transformer, with one design choice for each requirement above. MoREGT represents both rows and cells as tokens, so a row token can collect information from its own cells and exchange it with the rows it links to. Attention between two rows incorporates the difference between their log-ages and the name of the foreign-key relationship that connects them. To process columns whose meaning varies across databases, MoREGT replaces shared feed-forward layers with mixtures of experts, motivated by evidence that, in a stylized setting, a mixture of experts learns a problem composed of distinct subproblems on which a dense network fails [17]. To enable semantically related columns across databases to share experts, they are selected using a value-free description of each cell, consisting of its column-name embedding, data type and recency. Routing therefore depends on semantics of the data (column, type and age), not on its value or hidden state. No parameter depends on a schema: per-database quantities such as column statistics and name embeddings are frozen buffers, so a single checkpoint transfers to unseen databases.

To supply target-specific structure, MoREGT is paired with an agentic loop: a label-blind LLM agent reads the target schema, the task description and column statistics and proposes validated derived columns for a given task, each a single-row formula such as a rate, a numerical band or a group of codes. The agent never accesses labels, task tables or model scores, and because each derived column reads a single row, it is exposed to the model only when its inputs are temporally visible. Unlike the task-specific aggregations over linked tables that RDL was designed to replace [7], these columns leave all aggregation across rows, tables and time to the model.

Under the same leave-one-database-out pretraining as RT, the MoREGT backbone alone improves on RT by 2.25 in mean AUROC and 5.83 in mean nMAE. With the agent-derived columns added, MoREGT achieves the best mean scores for relational foundation models on the RelBench leaderboard, 79.63 AUROC and 26.98 nMAE, compared with 79.02 and 27.11 reported for RT-J [21] which is trained on a dataset with 400x more databases compared to what MoREGT is trained on. MoREGT also matches KumoRFM-2 [14] on classification and improves on it and on KumoRelational on regression. Our contributions are:

  • Transferable relational representation learning. We introduce MoREGT, a graph transformer with explicit row and cell tokens, time- and relationship-aware attention, value-free expert routing, and a relational pretraining objective designed to learn structure that transfers across schemas.

  • Label-blind database adaptation. We develop an LLM agent that generates validated, single-row derived columns from the target database’s schema and statistics, without access to task labels or model feedback.

  • Data-efficient cross-schema generalization. With derived columns, MoREGT achieves the best mean classification and regression performance among the compared methods on RelBench with a smaller pretraining dataset, suggesting that transferable representations and target-specific adaptation can partly substitute for larger pretraining corpora.

Related Work

Relational deep learning and foundation models. Relational deep learning treats a database as a temporal graph with one node per row and one edge per foreign-key link [7], and RelBench provides its standard tasks [22]. Models such as RelGNN [1] and RelGT [5] build on heterogeneous graph networks [12] and are trained for one database, with parameters tied to its node and edge types; RelGT also gives each node a time token. Foundation models transfer across schemas instead. For single tables, CARTE embeds each cell together with its column name using a language model [18]. For relational databases, Griffin pretrains a graph-centric model [26], RT tokenizes every cell with its column name and attends along columns, rows and key links [20], KumoRFM predicts by in-context learning [14], and Plural and RT-J scale RT’s pretraining data [19, 21]. MoREGT keeps RT’s schema-free cell tokens and adds row tokens, attention that depends on log-age differences and foreign-key names, experts routed on column meaning, and an objective that masks additional cells of the seed row so that its properties must be recovered from relational context. RT-J also reports experiments with multi-cell masking [21].

Mixture of experts. Sparse mixtures of experts add parameters at a fixed cost per token [24, 6], often alongside shared experts that process every token [3], and theory shows that experts can split a task into subproblems that a dense network learns poorly [17]. Most routers read the token’s hidden state, while hash layers route by token identity [23] and DEMix by a document’s known domain [9]. Our router is likewise fixed by what the input is, a value-free signature of each cell, but unlike a token identity or a domain label, this signature is defined for columns never seen in pretraining. Because column types are long-tailed, we also replace load balancing with an orthogonality penalty on the router.

Feature engineering and data-science agents. Deep feature synthesis enumerates aggregations and transformations along foreign keys [16]. LLM-based methods add the meaning of columns but select features with labels: CAAFE keeps a generated feature only if it improves validation performance [11], and FeatLLM derives rules from labelled examples [10]. RelAgent [13] brings this to relational databases: an LLM agent writes multi-table SQL feature queries with temporal filters, revises them using validation scores, and trains a tree ensemble that then predicts without the LLM. Our agent sees no labels, task table or model scores, and writes only single-row formulas, which keeps derived columns free of future information by construction rather than through query filters. Its columns are added to the database for a pretrained relational model rather than replacing that model.

MoREGT

MoREGT predicts a property of an entity in a relational database from its temporal relational neighborhood Figure 2. After formalizing the setting, we describe the two parts that correspond to the two challenges of § 1: a schema-transferable row–cell graph transformer (§ 3.1–§ 3.4) and a label-blind agent that adds target-specific derived columns (§ 3.5).

Figure 2. Overview of MoREGT. (a) Temporal neighborhood of a seed e∗e^{*} at time t∗t^{*}. (b) Cell and row tokens. (c) Two-level relational block, repeated LL times. (d) Stage 1 pretrains on the other databases; stage 2 fine-tunes on the target database with agent-derived columns (green). (e) Expert routing reads what a cell is, never its value. (f) Attention depends on log-age and on the name of each FK role.

Problem setup. A relational database D\mathcal{D} is a set of tables linked by primary–foreign key (PK–FK) relationships [7, 22]. We index rows by ii and columns by jj. Row ii belongs to table T(i)T(i) and carries a timestamp tit_i if that table is timed, each feature column jj has a semantic type st⁡(j)\operatorname{st}(j) (numerical, categorical, text or timestamp), and for a feature column jj of T(i)T(i), cell (i,j)(i,j) holds the value xi,jx_{i,j}. Following RDL, we view D\mathcal{D} as a heterogeneous temporal graph with one node per row. Every FK column col of a child table CC that references a parent table PP defines a role ρ=(C,col,P)\rho=(C,\mathrm{col},P) and a pair of typed edges in both directions, so two FKs into the same parent, e.g., buyer_id and seller_id, remain distinct. Key columns enter the model only as graph structure, never as cell values. A task specifies seeds (e∗,t∗)(e^*,t^*), an entity row and a prediction time, and asks for a label yy describing e∗e^* at t∗t^*. For each seed, a temporal sampler expands NhopN_{\mathrm{hop}} hops from e∗e^* and keeps, per edge type and frontier node, the NnbrN_{\mathrm{nbr}} most recent neighbors with ti≤t∗t_i \le t^*; rows of untimed tables are not time-filtered. We write Δi=max⁡(0,t∗−ti)\Delta_i=\max(0,t^*-t_i) for the age of row ii at t∗t^*, in seconds, and τi=log⁡(1+Δi)\tau_i=\log(1+\Delta_i) for its log-age. The time features φi∈R2K+1\boldsymbol{\varphi}_i \in\mathbb{R}^{2K+1} of row ii are sin⁡(ωfτi)\sin(\omega_f\tau_i) and cos⁡(ωfτi)\cos(\omega_f\tau_i) for KK fixed frequencies ωf\omega_f spaced logarithmically in [ωmin⁡,ωmax⁡][\omega_{\min},\omega_{\max}], plus a flag that marks a clamped time (Appendix § 6); the sine and cosine terms are set to zero for untimed rows so that an unknown age differs from Δi=0\Delta_i=0.

Row–Cell Relational Representation

Cell and row embeddings. Following RT [20], every cell of every sampled row becomes one token (Figure 2b). The seed row comes first, cells within a row are ordered by column name, and sequences are capped at NcellN_{\mathrm{cell}} cells. An encoder shared by all columns of a semantic type maps the value xi,jx_{i,j} to a vector vi,j\mathbf{v}_{i,j}; in particular, a categorical value cc is encoded through the frozen text embedding nj,c\mathbf{n}_{j,c} of “table TT, column jj, value cc”, where TT is the table of jj (Appendix § 6 gives all encoders). The initial embedding of cell (i,j)(i,j), which the first block reads, adds a value component and a schema component,

Hi,j(0)=Wvalvi,j+bval+Wnamenj+bname,(1)\mathbf{H}^{(0)}_{i,j}=\mathbf{W}_{\mathrm{val}}\mathbf{v}_{i,j}+\mathbf{b}_{\mathrm{val}}+\mathbf{W}_{\mathrm{name}}\mathbf{n}_j+\mathbf{b}_{\mathrm{name}}, \tag*{(1)}

where nj∈Rdn\mathbf{n}_j \in\mathbb{R}^{d_n} is the frozen embedding of “table TT, column jj”, and Wval\mathbf{W}_{\mathrm{val}}, bval\mathbf{b}_{\mathrm{val}}, Wname\mathbf{W}_{\mathrm{name}} and bname\mathbf{b}_{\mathrm{name}} are learned. MoREGT also keeps one embedding per sampled row, so that a row, and not only its cells, can take part in attention and be read by the prediction head. We write Ui(0)\mathbf{U}^{(0)}_i for the initial embedding of row ii (Appendix § 6).

Overview. The backbone maps the initial embeddings of one example to a prediction for its seed row. It keeps one embedding per cell, which carries content, and one per sampled row, which carries relational structure. We write H(ℓ)\mathbf{H}^{(\ell)} for the cell embeddings Hi,j(ℓ)\mathbf{H}^{(\ell)}_{i,j} and U(ℓ)\mathbf{U}^{(\ell)} for the row embeddings Ui(ℓ)\mathbf{U}^{(\ell)}_i after ℓ\ell blocks, all in Rd\mathbb{R}^d. Besides the embeddings, every block reads a context G\mathcal{G} that is fixed for the example and contains no cell value: the log-ages τ=(τi)i\boldsymbol{\tau}=(\tau_i)_i, the set E\mathcal{E} of sampled FK edges with the role ρ(i,i′)\rho(i,i') of each edge, the column-name embeddings nj\mathbf{n}_j, and the value-free signatures Z\mathbf{Z} and S\mathbf{S} defined in § 3.3. The backbone applies LL blocks and predicts from the final embedding of the seed row:

(H(ℓ),U(ℓ))=MoREGTBlock⁡(ℓ)(H(ℓ−1),U(ℓ−1);G),y^=MLP⁡(LN⁡(Ue∗(L))),(2)\begin{aligned} (\mathbf{H}^{(\ell)},\mathbf{U}^{(\ell)}) &= \operatorname{MoREGTBlock}^{(\ell)}\left(\mathbf{H}^{(\ell-1)},\mathbf{U}^{(\ell-1)};\mathcal{G}\right),\\ \hat{y} &= \operatorname{MLP}\left(\operatorname{LN}\left(\mathbf{U}^{(L)}_{e^*}\right)\right), \tag*{(2)} \end{aligned}

where ℓ=1,…,L\ell=1,\ldots,L, each block has its own parameters, and LN is LayerNorm.

Inside a block. MoREGTBlock⁡(ℓ)\operatorname{MoREGTBlock}^{(\ell)} takes the cell embeddings H(ℓ−1)\mathbf{H}^{(\ell-1)}, the row embeddings U(ℓ−1)\mathbf{U}^{(\ell-1)} and the context G\mathcal{G}, and returns new embeddings H(ℓ)\mathbf{H}^{(\ell)} and U(ℓ)\mathbf{U}^{(\ell)} of the same sizes. It applies the six sublayers numbered in Figure 2c: two update the cells, one pools the cells into their rows, two update the rows, and one broadcasts the rows back to their cells:

H′=H(ℓ−1)+CellAttn⁡(RMSNorm⁡(H(ℓ−1));τ),H′′=H′+CellMoE⁡(RMSNorm⁡(H′);Z),U′=U(ℓ−1)+Pool⁡(U(ℓ−1),H′′;S,{nj}),U′′=U′+RowAttn⁡(RMSNorm⁡(U′);τ,E),U(ℓ)=U′′+RowMoE⁡(RMSNorm⁡(U′′);S),Hi,j(ℓ)=Hi,j′′+WbUi(ℓ),(3)\begin{aligned} \mathbf{H}' &= \mathbf{H}^{(\ell-1)} + \operatorname{CellAttn}\left(\operatorname{RMSNorm}\left(\mathbf{H}^{(\ell-1)}\right); \boldsymbol{\tau}\right),\\ \mathbf{H}'' &= \mathbf{H}' + \operatorname{CellMoE}\left(\operatorname{RMSNorm}\left(\mathbf{H}'\right); \mathbf{Z}\right),\\ \mathbf{U}' &= \mathbf{U}^{(\ell-1)} + \operatorname{Pool}\left(\mathbf{U}^{(\ell-1)}, \mathbf{H}''; \mathbf{S}, \{\mathbf{n}_j\}\right),\\ \mathbf{U}'' &= \mathbf{U}' + \operatorname{RowAttn}\left(\operatorname{RMSNorm}\left(\mathbf{U}'\right); \boldsymbol{\tau}, \mathcal{E}\right),\\ \mathbf{U}^{(\ell)} &= \mathbf{U}'' + \operatorname{RowMoE}\left(\operatorname{RMSNorm}\left(\mathbf{U}''\right); \mathbf{S}\right),\\ \mathbf{H}_{i,j}^{(\ell)} &= \mathbf{H}_{i,j}'' + \mathbf{W}_{b}\mathbf{U}_{i}^{(\ell)}, \tag*{(3)} \end{aligned}

where the arguments after the semicolons are the parts of G\mathcal{G} that each sublayer reads, RMSNorm is applied to each embedding separately, and Wb\mathbf{W}_{b} is learned. Every update is residual. The attention and expert sublayers are pre-norm, while the two transfers between levels, pooling and broadcast, read the embeddings as they are. Three components are specific to MoREGT: attention scores that depend on the log-ages of the rows, a row-attention bias computed from the name of the FK role that links two rows (both in § 3.2 and Figure 2f), and experts routed on value-free signatures (§ 3.3 and Figure 2e). We group the sublayers by their function below, while Figure 2c gives their execution order; Appendix § 6 gives the remaining specifications.

❸ Pool into rows. Row attention moves information along FK edges, and FK edges link rows, not cells, so before that step each row needs a summary of its own cells as they stand after the cell sublayers. Pool computes this summary by multi-head cross-attention from each row to its own cells. The queries come from the row’s embedding and signature [Ui(ℓ−1);Si][\mathbf{U}_{i}^{(\ell-1)}; \mathbf{S}_i], the keys from the frozen column-name embeddings nj\mathbf{n}_j, and the values from the cell embeddings Hi,j′′\mathbf{H}_{i,j}''. Because the keys are column names rather than cell embeddings, each head selects cells by what their columns mean, regardless of their values. Because the names come from a frozen encoder, the columns of an unseen database receive keys in the same space as the pretraining columns.

❻ Broadcast to cells. The last line of Eq. (3) returns what each row gathered along FK edges to its cells, adding a projection of the row embedding to every cell of the row. The next block’s cell attention, and after block LL the pretraining heads, read these cell embeddings.

Unlike RT, which represents rows only through their cells, MoREGT retains row tokens for FK-level message passing and downstream prediction.

Time- and Relationship-Aware Attention

❶ Cell attention. Whether a related row informs a prediction at t∗t^{*} can depend on how long before t∗t^{*} it occurred relative to other rows, and a pretrained model must apply to databases that cover very different time spans. RT embeds a timestamp as a cell value standardized over the whole database ([20]), so the time between two rows must be inferred by comparing two values whose scale is set by that span. MoREGT instead puts the log-ages of the two rows into every attention score. CellAttn is multi-head attention among all cells of the example with no structural mask, so content mixes freely across rows and tables. Following rotary embeddings ([25]), KK pairs of dimensions of the query and key of every token of row ii are rotated by the angles ωfτi\omega_f\tau_i, f=1,…,Kf=1,\ldots,K, with the frequencies of φi\varphi_i. Rotating a query by one angle and a key by another leaves only the difference of the angles in their dot product, so in one head the score of cell p=(i,j)p=(i,j) for cell p′=(i′,j′)p'=(i',j'), which CellAttn uses between cells and RowAttn uses between rows, is

σpp′=1dh(qˉpTkp′+∑f=1Kqp[f]TR(ωf(τi′−τi))kp′[f]),(4)\sigma_{pp'} = \frac{1}{\sqrt{d_h}}\left(\bar{\mathbf{q}}_p^{\mathsf{T}}\mathbf{k}_{p'} + \sum_{f=1}^{K}\mathbf{q}_p[f]^{\mathsf{T}}\mathbf{R}\left(\omega_f(\tau_{i'}-\tau_i)\right)\mathbf{k}_{p'}[f]\right), \tag*{(4)}

where dhd_h is the head width, qˉp\bar{\mathbf{q}}_p and kˉp′\bar{\mathbf{k}}_{p'} are the unrotated parts of the query of pp and the key of p′p', qp[f],kp′[f]∈R2\mathbf{q}_p[f],\mathbf{k}_{p'}[f] \in\mathbb{R}^2 are their ff-th rotated pairs, and R(θ)\mathbf{R}(\theta) is the planar rotation by the angle θ\theta. Two learned offsets mark pairs in which a row is untimed or its time was clamped (Appendix § 6). Otherwise time enters the score only through τi′−τi=log⁡1+Δi′1+Δi\tau'_i-\tau_i=\log\frac{1+\Delta'_i}{1+\Delta_i}. Because ages are measured in seconds, the +1 is negligible and this difference is the log-ratio of the two ages: one day versus two days is as far apart as one year versus two years (both ≈log⁡2\approx\log2), whatever time span the database covers. Cells carry no row identifier, so in the first block cells of rows with equal ages are not told apart by row; from the second block on, broadcast ❻ adds each row’s embedding to its cells.

❹ Row attention. Rows linked through different FKs play different parts: the buyer and the seller of an item are both users but relate to the item differently. The RDL baseline (Fey et al., [7]) passes messages with separate parameters per edge type, and such parameters do not exist for an edge type unseen in pretraining. MoREGT instead computes an attention bias from the name of the role that links two rows (Figure 2f). RowAttn is multi-head attention in which row ii attends to itself and to the set N(i)={i′:(i,i′)∈E}\mathcal{N}(i)=\{i':(i,i')\in E\} of rows linked to it by a sampled FK edge. In each head mm, the score of row ii for row i′i' is the time-aware score of Eq. (4), computed from row queries and keys, plus a bias that, for i′∈N(i)∪{i}i'\in\mathcal{N}(i)\cup\{i\}, is

βii′m=amTWattnρ(i,i′)role+bm,dir⁡(i,i′),(5)\beta_{ii'}^{m}=\mathbf{a}_m^{\mathsf{T}}\mathbf{W}_{\mathrm{att}}\mathbf{n}^{\mathrm{role}}_{\rho(i,i')}+b_{m,\operatorname{dir}(i,i')}, \tag*{(5)}

where nρ(i,i′)role\mathbf{n}^{\mathrm{role}}_{\rho(i,i')} is the frozen embedding of “table CC column col references table PP” for the role ρ(i,i′)=(C,col,P)\rho(i,i')=(C,\mathrm{col},P) of the edge, Watt\mathbf{W}_{\mathrm{att}} is a projection shared by the heads, am\mathbf{a}_m and bm,⋅b_{m,\cdot} are learned per head, and dir⁡(i,i′)\operatorname{dir}(i,i') is parent if row ii holds the FK that references i′i', child if i′i' holds the FK that references ii, and self if i′=ii'=i, in which case the role term is zero. Because each FK column defines its own role, the buyer and the seller of an item receive different biases. Because the bias is computed from the role’s name rather than learned per role, a role unseen in pretraining receives a bias close to that of similarly named roles.

Unlike RT, which compares standardized timestamp values (Ranjan et al., [20]), and RDL models, which learn separate parameters per edge type (Fey et al., [7]), MoREGT puts relative log-age into both cell and row attention and derives edge-specific biases from FK-role names without schema-specific parameters.

Value-Free Mixture-of-Relational-Experts

Value-free signatures. A signature carries the semantics of what a cell or row is. Signatures are the only input of the expert routers (Eq. (8)), so they decide which parameters process each cell and row. Because they hold no values, this decision is fixed by the schema and by age, and because they are built from frozen name embeddings, a new database receives signatures in the same space as the pretraining databases. Both kinds encode age through the time features φi\boldsymbol{\varphi}_i of row ii defined in the problem setup. The signature of cell (i,j)(i,j) combines the meaning of its column, its semantic type and the age of its row, so cells of one column can take different routes in recent and old rows:

Zi,j=RMSNorm⁡(Wznj+Est[st⁡(j)]+Wτφi),(6)\mathbf{Z}_{i,j}=\operatorname{RMSNorm}\left(\mathbf{W}_z\mathbf{n}_j+\mathbf{E}_{\mathrm{st}}[\operatorname{st}(j)]+\mathbf{W}_{\tau}\boldsymbol{\varphi}_i\right), \tag*{(6)}

where Zi,j∈Rds\mathbf{Z}_{i,j}\in\mathbb{R}^{d_s}, Wz\mathbf{W}_z and Wτ\mathbf{W}_{\tau} are learned projections, and Est\mathbf{E}_{\mathrm{st}} holds one learned embedding per semantic type. The signature of row ii says what kind of row it is and how it relates to the seed, through its table, the FK role through which it is first reached from the seed, its hop distance and its age:

Si=RMSNorm⁡(WTnT(i)tab+Wρnρ0(i)role+Ehop[hop⁡(i)]+Wτ′φi),(7)\mathbf{S}_i=\operatorname{RMSNorm}\left(\mathbf{W}_{T}\mathbf{n}^{\mathrm{tab}}_{T(i)}+\mathbf{W}_{\rho}\mathbf{n}^{\mathrm{role}}_{\rho_0(i)}+\mathbf{E}_{\mathrm{hop}}[\operatorname{hop}(i)]+\mathbf{W}'_{\tau}\boldsymbol{\varphi}_i\right), \tag*{(7)}

where nTtab\mathbf{n}^{\mathrm{tab}}_{T} is the frozen embedding of the name of table TT, nρrole\mathbf{n}^{\mathrm{role}}_{\rho} is the role embedding of Eq. (5), ρ0(i)\rho_{0}(i) is the role through which row ii is first reached (the role term is zero for the seed row), Ehop\mathbf{E}_{\mathrm{hop}} holds one learned embedding per hop, and WT\mathbf{W}_{T}, Wρ\mathbf{W}_{\rho} and Wτ′\mathbf{W}'_{\tau} are learned projections.

❷ Cell experts. CellMoE is a mixture of experts whose router reads only the signature Zi,j\mathbf{Z}_{i,j}, while the experts transform the normalized cell embedding H^i,j=RMSNorm⁡(Hi,j′)\hat{\mathbf{H}}_{i,j} = \operatorname{RMSNorm}(\mathbf{H}'_{i,j}) (Figure 2e). Its output is added to the cell embedding in the second line of Eq. (3):

CellMoE⁡(H^i,j;Zi,j)=∑e∈Ti,jgi,j,eEe(H^i,j)+∑e=1NsEesh(H^i,j),(8)\operatorname{CellMoE}(\hat{\mathbf{H}}_{i,j};\mathbf{Z}_{i,j}) = \sum_{e \in\mathcal{T}_{i,j}} g_{i,j,e} \mathbf{E}_{e}(\hat{\mathbf{H}}_{i,j}) + \sum_{e=1}^{N_{\mathrm{s}}} \mathbf{E}^{\mathrm{sh}}_{e}(\hat{\mathbf{H}}_{i,j}), \tag*{(8)}
gi,j=softmax⁡(TopK⁡kr(WgZi,j)),\mathbf{g}_{i,j} = \operatorname{softmax}\left(\operatorname{TopK}_{k_{\mathrm{r}}}(\mathbf{W}_{\mathrm{g}}\mathbf{Z}_{i,j})\right),

where Wg\mathbf{W}_{\mathrm{g}} is the router, which scores the NrN_{\mathrm{r}} routed experts Ee\mathbf{E}_{e}; TopK⁡kr\operatorname{TopK}_{k_{\mathrm{r}}} keeps the krk_{\mathrm{r}} largest scores and sets the others to −∞-\infty; Ti,j\mathcal{T}_{i,j} is the set of experts it keeps; and E1sh,…,ENssh\mathbf{E}^{\mathrm{sh}}_{1}, \ldots, \mathbf{E}^{\mathrm{sh}}_{N_{\mathrm{s}}} are always-active shared experts [3]. All experts are feed-forward networks. The cell router is regularized only by an orthogonality penalty on its expert directions, with no load-balancing loss (Appendix § 6).

We route on signatures because a dense feed-forward layer shared by every column of every database, as in RT (Ranjan et al., [20]), must fit one pooled problem, which each database added to pretraining makes more heterogeneous. For data made of distinct subproblems, a dense network trained by gradient descent can fail to find this structure, while a mixture of experts splits the task and learns each subproblem (Kawata et al., [17]). In MoREGT the subproblems are observed rather than latent: they are given by the column, type and age of each cell. The router therefore reads what a cell is, and the experts transform what it holds after attention has mixed in context. A routed expert’s parameters enter the sublayer only at the cells routed to it, so it is trained only on those cells, and this assignment depends neither on values nor on which neighbors were sampled. For the same reason, the route cannot encode the label.

❺ Row experts. RowMoE applies Eq. (8) to the normalized row embeddings, with its own experts and a cosine router (Chi et al., [2]) on the row signature Si\mathbf{S}_{i}, so that rows of different tables, roles and hops can use different experts. Its router is regularized by the same orthogonality penalty and, unlike the cell router, by a load-balancing term (Appendix § 6).

Unlike conventional content-dependent MoE routing ([24, 6]), expert assignment uses schema metadata and age, allowing related columns in an unseen database to select the same pretrained experts.

Relational Pretraining and Transfer

Relational pretraining objective. Every table with a maskable column (numerical with nonzero variance, or categorical with at least two classes) supplies seeds; a timed row uses its own timestamp as t∗t^{*}, and a static row draws t∗t^{*} from the training window. RT masks a single target cell and predicts it from its context (Ranjan et al., [20]), which leaves the other cells of that row visible, so a column that is close to a function of its siblings can be recovered without the relational context. We mask one maskable cell of the seed row, as in RT, and each other maskable cell of the seed row with probability pmaskp_{\mathrm{mask}}, and we never mask cells of context rows. The objective thus has the form of a downstream task, predicting a property of an entity at t∗t^{*} from its relational past. Numerical targets are regressed with a Huber loss from H^i,j=RMSNorm⁡(Hi,j(L))\hat{\mathbf{H}}_{i,j} = \operatorname{RMSNorm}(\mathbf{H}^{(L)}_{i,j}), and categorical targets are scored against the column’s own frozen category embeddings, oi,j,c=(WcatH^i,j+bcat)Tnj,co_{i,j,c} = (\mathbf{W}_{\mathrm{cat}}\hat{\mathbf{H}}_{i,j} + \mathbf{b}_{\mathrm{cat}})^{\mathsf{T}}\mathbf{n}_{j,c}, so the output space has no schema-specific parameters either. The pretraining loss is Lnum+λcatLcat+λauxLaux\mathcal{L}_{\mathrm{num}} + \lambda_{\mathrm{cat}}\mathcal{L}_{\mathrm{cat}} + \lambda_{\mathrm{aux}}\mathcal{L}_{\mathrm{aux}}, where Laux\mathcal{L}_{\mathrm{aux}} collects the routing regularizers of both levels over all blocks (Appendix § 6).

Unlike single-cell reconstruction in RT (Ranjan et al., [20]), additional seed-row masking reduces the opportunity to reconstruct a target from its sibling cells alone. RT-J also reports experiments with multi-cell masking (Ranjan et al., [21]); in MoREGT the extra masking is confined to the seed row, and context rows are never masked.

Transfer to unseen databases. No parameter of MoREGT has a shape that depends on a database: every per-database quantity (column statistics and name, category and role embeddings) is a frozen buffer rebuilt for each database, so one checkpoint applies to any schema. For each RelBench database D\mathcal{D}, we pretrain a trunk θ(−D)\theta^{(-\mathcal{D})} on all other databases of the benchmark, so it has never seen a row, a column name or the schema of D\mathcal{D}; all trunks share one recipe and use no rows past the validation timestamp. Fine-tuning on a task of D\mathcal{D} starts from θ(−D)\theta^{(-\mathcal{D})}, replaces the reconstruction heads by the prediction head of Eq. (2), and trains all parameters with binary cross-entropy or, for regression, an ℓ1\ell_1 loss on standardized targets, whose minimizer, the conditional median, is MAE-optimal; in both cases the loss includes λauxLaux\lambda_{\mathrm{aux}}\mathcal{L}_{\mathrm{aux}}. Appendix § 6 gives all hyperparameters and the optimization and model-selection procedures.

As in RT (Ranjan et al., [20]), and unlike RDL models whose parameters are tied to one database’s node and edge types ([1, 5]), this two-stage procedure retains a single schema-independent parameterization: pretraining learns transferable relational structure, while fine-tuning adapts that structure to an unseen database.

Label-Blind Agentic Loop for Derived Columns

Why derived columns, and why an agent. Some quantities that matter for a task are conventions of one database: in Formula 1, status codes 5 to 10 denote mechanical failures, and race time divided by completed laps gives a driver’s pace. Each takes one line to write, but code groups and bands are irregular functions that neural networks learn poorly ([8]), and rates are interactions between columns [15]. A derived column hands such a quantity to the model as one extra cell per row. Derived columns are not free. Each is a function of its own row, so it adds no information about the target beyond that row, and each costs one token per sampled row under the cap NcellN_{\mathrm{cell}} (Appendix A). Useful columns are therefore few and depend on what the columns mean, which favors an LLM agent that reads the schema over enumerating transformations [16].

A label-blind agent. We use a label-blind LLM agent (Figure 3) that receives the database name and a one-line task description; its only action is a read-only profiling tool that returns schemas, column profiles (null rate, cardinality, quantiles, moments and frequent values) and key candidates inferred from the data. All declared primary keys, foreign keys and time columns are hidden; the agent never sees the task table, labels or model scores. This enables agent columns to be produced before any labels exist. Each tool call is parsed from the transcript, and any run with a call outside the allowlist is discarded.

Figure 3. Agentic loop for derived columns on rel-f1. A label-blind LLM agent profiles the data, proposes row-local formulas and revises them from screening feedback. Accepted formulas become new columns, used only in stage 2 (Figure 2d). The proposals shown are illustrative.

Definition 3.1 (Row-local expression). A row-local expression on table TT is built from the columns of TT, numeric constants, arithmetic operators (including floor division, modulo and powers), unary signs, comparisons (which evaluate to 0 or 1), and the element-wise functions abs, log, log1p, ⋅\sqrt{\cdot}, min and max. Attribute access, subscripts, Boolean connectives, aggregation, grouping, sorting, shifting, and references to other tables or other proposals are excluded.

Screening and feedback. The agent proposes named row-local expressions xi,j=fj(xi,j1,…,xi,jm)x_{i,j}=f_j(x_{i,j_1},\ldots,x_{i,j_m}), each defining a new column jj and coming with a description, units, a claimed range and a rationale. A candidate is rejected if it fails to parse or evaluate, references a missing or forbidden column, is constant, infinite or entirely missing, has a missing rate more than η\eta above that of its inputs, places less than a fraction qq of its values inside its own claimed range, or has absolute correlation above rmax⁡r_{\max} with any numerical column of its table, including earlier proposals. Rejection reasons are returned to the agent for up to RR rounds, and the feedback contains no model score. Accepted expressions are evaluated over their tables, missing and infinite results are set to 0, and each result becomes an ordinary numerical column with its own statistics and name embedding, which MoREGT processes like any native column.

Temporal visibility by construction. The sampler admits a timed row only if ti≤t∗t_i \le t^*. A derived value is a function of one row, so it is visible to the model exactly when its inputs are. Aggregation, shifting and cross-table references, which could read rows from after t∗t^*, are excluded from the grammar for this reason.

Unlike label-guided feature search [11, 10, 13] or multi-table aggregations [16], this adaptation requires no downstream labels or model feedback and preserves row-local temporal visibility.

Experiments

In this section, we first describe the experimental setup used to train MoREGT. We then outline baselines followed by results and ablation studies.

Experimental setup. We evaluate on the 21 entity tasks of the seven RelBench databases [22]: 12 binary classification tasks scored by AUROC and 9 regression tasks scored by nMAE (MAE divided by the standard deviation of the training labels). We use the official temporal train/validation/test splits and report test scores. Pretraining is leave-one-database-out: for each target database, a 134M-parameter MoREGT (10 blocks, width 256, mixture-of-experts feed-forward layers with 8 cell-level and 4 row-level experts) is pretrained with masked-cell prediction on the other six databases, so pretraining never sees the target database. Column names are embedded once with

Baselines. We compare against three groups of methods. Trained from scratch: RelGT (Dwivedi et al. [5]) and GelGT (Ding et al. [4]), graph transformers trained separately on each task without pretraining. Pretrained and fine-tuned: the Relational Transformer (RT) (Ranjan et al. [20]), which uses the same cell-token input as our model and is pretrained on the other RelBench databases before fine-tuning on the target task, making it the most direct comparison to MoREGT (w/o agent); and RT-J and RT-PluRel, RT variants with larger pretraining corpora (Ranjan et al. [21]; Kothapalli et al. [19]). In-context: KumoRelational and KumoRFM-2 (Hudovernik et al. [14]), relational foundation models that predict from labelled examples in context, without task-specific training. RT results are copied from Ranjan et al. [20] where reported and otherwise reproduced from its released checkpoints. KumoRFM-2 results are taken from Hudovernik et al. [14], with its published MAE converted to nMAE; all other baselines come from the RelBench leaderboard.

Results

Table 1 and Table 2 compare MoREGT with graph transformers trained from scratch (RelGT, GelGT), pretrained and fine-tuned relational transformers (RT, RT-J, RT-PluRel), and in-context relational foundation models (KumoRelational, KumoRFM-2). MoREGT (with agent) attains the best mean on both: 79.63 AUROC on the 12 classification tasks, against 79.60 for KumoRFM-2 and 79.02 for RT-J, and 26.98% nMAE on the 9 regression tasks, against 27.11% for RT-J and 29.13% for KumoRFM-2. Its pretraining covers only the six RelBench databases other than the target, compared with roughly 2,400 for RT-J. The lead comes from consistency rather than per-task wins. MoREGT (with agent) is best or tied for best on 4 of the 21 tasks (hm user-churn, both stack classification tasks, f1 driver-position) and within one point of the best on the f1 and trial classification tasks. On regression, its largest deficit to the best baseline (6.4 nMAE on trial site-success) is smaller than that of any other method (8.1 for RT-J, 22.6 for KumoRFM-2). It trails the best baseline by 1–3 points on amazon and avito, and by 4.5 AUROC on event user-ignore. Without agent-derived columns, MoREGT (w/o agent) (78.45 AUROC / 27.91% nMAE) outperforms both from-scratch baselines and RT. The classification gap to KumoRFM-2 (0.03 AUROC) is within seed variation; the two models differ chiefly in setting, since KumoRFM-2 predicts in-context without task training while MoREGT is fine-tuned per task. Appendix Figure A1 reports inference time.

DatasetTaskRelGTGelGTRTRT-JRT-PluRelKumoRel.KumoRFM-2‡MoREGT (w/o agent)MoREGT (with agent)
amazonitem-churn82.581.983.483.283.282.582.281.36±0.3881.71±0.22
user-churn70.469.570.871.5<u>71.3</u>69.569.169.44±0.3170.17±0.15
avitouser-clicks<u>68.3</u>68.459.066.865.968.167.464.69±2.6267.79±1.37
user-visits66.8<u>67.1</u>65.266.966.867.069.465.26±1.5066.65±0.36
eventuser-ignore81.6<u>88.4</u>72.485.680.985.190.885.67±5.7686.31±3.38
user-repeat76.184.169.979.880.381.681.7<u>82.84±1.94</u>82.63±2.42
f1driver-dnf75.975.684.282.882.1<u>84.3</u>84.683.90±0.2584.11±0.39
driver-top383.585.991.991.790.090.8<u>92.2</u>92.27±0.9292.06±0.76
hmuser-churn69.368.270.570.5<u>70.6</u>69.969.368.13±0.3570.82±0.06
stackuser-badge86.387.588.789.2<u>89.3</u>88.187.288.31±0.2889.40±0.12
user-engagement90.5<u>90.7</u>90.290.690.990.289.490.65±0.3790.90±0.14
trialstudy-outcome68.672.568.269.671.274.072.068.93±5.56<u>73.05±1.72</u>
Mean76.6578.3276.2079.0278.5479.26<u>79.60</u>78.4579.63

Table 1. Entity classification on RelBench: TEST AUROC (%; higher is better). Bold: best; <u>underline</u>: second best. ‡From Hudovernik et al. [14]. RT reproduced with checkpoints from Ranjan et al. [20], other baselines from the RelBench leaderboard.

a frozen Qwen3-Embedding-4B and text values with MiniLM. Each fold is pretrained on 4 NVIDIA H100 GPUs for 18 hours (batch size 64, sequence length 1,024, learning rate 3×10−43 \times10^{-4}), keeping the checkpoint with the best validation masked-cell score; Appendix Table A1 lists all hyperparameters. For each task we fine-tune the full model on one H100 with past labels given as context, following RT. We fine-tune the pre-trained model for 3 seeds and early stopping on validation. For MoREGT (with agent), an LLM agent (Claude Opus 5) also proposes 8–14 derived columns per task (Appendix Table A2), stating each column’s expected units and range before its values are computed. The code is available at https://anonymous.4open.science/r/MoREGT-7BDC/README.md.

DatasetTaskRelGTGelGTRTRT-JRT-PluRelKumoRel.KumoRFM-2‡MoREGT (w/o agent)MoREGT (with agent)
amazonitem-ltv8.38.312.47.2<u>7.3</u>7.58.07.95±0.187.64±0.10
user-ltv24.825.227.3<u>24.3</u>24.625.024.225.91±0.3625.02±0.18
avitoad-ctr36.1<u>35.3</u>41.936.037.933.235.535.49±0.2335.52±0.43
eventuser-attendance32.730.656.631.631.631.7<u>30.7</u>30.92±0.2631.09±0.62
f1driver-position55.853.3<u>39.7</u>37.7<u>39.7</u>43.740.637.67±1.20<u>37.70±1.98</u>
hmitem-sales10.811.213.37.6<u>7.5</u>8.06.98.27±0.287.92±0.13
stackpost-votes<u>12.8</u>12.914.9<u>12.8</u>13.612.512.514.15±1.3813.30±0.00
trialsite-success<u>68.6</u>68.483.676.575.285.291.077.64±6.0174.84±5.56
study-adverse13.012.114.010.310.79.412.813.21±0.13<u>9.75±0.27</u>
Mean29.2028.5933.74<u>27.11</u>27.5728.4729.1327.9126.98

Table 2. Entity regression on RelBench: TEST nMAE = MAE/σtrain\sigma_{\mathrm{train}} (%; lower is better). Bold: best; <u>underline</u>: second best. ‡From Hudovernik et al. [14]; its published MAE converted to nMAE with the same train-label σ\sigma. RT reproduced with checkpoints from Ranjan et al. [20], other baselines from the RelBench leaderboard.

Component Ablations. To assess whether the architectural components benefit cross-database transfer, we ablate MoREGT under leave-one-database-out pretraining on rel-f1 (Table 3) by removing the introduced components. Each variant is pretrained from scratch under the same leave-one-database-out (LODO) protocol and fine-tuned with identical task data and agent-derived columns. The full model outperforms dense FFNs on all three tasks, improving AUROC by 0.64–1.66 points and reducing nMAE by 0.67 points. Replacing value-free routing with hidden-state routing, at comparable parameter count, also degrades all three tasks, supporting routing based on column semantics rather than token representations. Removing FK-role biases degrades two tasks, with a small improvement on driver-top3.

TaskFull modelDense FFNHidden routingNo FK-role bias
driver-dnf ↑84.11±0.3982.45±1.0782.82±0.6282.63±1.13
driver-top3 ↑92.06±0.7691.42±0.1592.00±0.2692.33±0.16
driver-position ↓37.70±1.9838.37±0.6340.16±2.2039.79±1.70

Table 3. Component ablation under leave-one-database-out pretraining. TEST AUROC for classification and nMAE for regression (%); mean ± std over 3 seeds.

Appendix Table A3 reports architecture ablations of MoREGT-small (30M parameters), trained from scratch on the nine RelBench tasks of rel-f1, rel-event and rel-trial. The full model has the best mean score on both classification (79.14% AUROC, 0.71 points above the next-best variant) and regression (44.40% nMAE, 1.89 points below the next-best variant), and it is best on five of nine tasks. The regression margin does not rest on the high-variance site-success task: without it, the full model still has the lowest mean nMAE (31.55%, against 32.48% for the next-best variant). The differences are clearest where variance across seeds is low. On study-adverse, every ablation raises nMAE by at least 1.93 points, and on driver-top3 every ablation lowers AUROC by at least 1.52 points.

FullDense FFNHidden routingNo time attn.No role bias
Classification (AUROC ↑\uparrow)
f1driver-dnf84.12±0.1384.12\pm0.1381.82±0.0981.82\pm0.0981.66±0.5481.66\pm0.5483.47±0.6583.47\pm0.6582.34±0.2282.34\pm0.22
driver-top393.04±0.7393.04\pm0.7391.03±0.6991.03\pm0.6991.52±0.2591.52\pm0.2590.97±0.4290.97\pm0.4291.16±0.3791.16\pm0.37
eventuser-repeat76.88±5.0076.88\pm5.0076.54±1.8476.54\pm1.8477.08±0.5277.08\pm0.5272.85±5.5172.85\pm5.5173.84±5.9673.84\pm5.96
user-ignore76.07±3.0776.07\pm3.0777.60±5.0677.60\pm5.0675.02±1.6175.02\pm1.6175.15±5.8675.15\pm5.8676.35±5.6576.35\pm5.65
trialstudy-outcome65.57±1.2965.57\pm1.2965.18±0.8465.18\pm0.8464.55±1.7364.55\pm1.7363.60±1.3863.60\pm1.3863.24±1.4063.24\pm1.40
Mean79.1478.4377.9777.2177.39
Regression (nMAE ↓\downarrow)
f1driver-position38.24±2.9538.24\pm2.9538.04±1.1638.04\pm1.1639.84±1.2939.84\pm1.2939.32±1.1639.32\pm1.1639.45±0.2039.45\pm0.20
eventuser-attendance43.43±6.5643.43\pm6.5645.36±5.0845.36\pm5.0843.34±3.9643.34\pm3.9643.06±2.1943.06\pm2.1946.29±6.2346.29\pm6.23
trialstudy-adverse12.99±0.2812.99\pm0.2814.92±0.0414.92\pm0.0415.02±0.1015.02\pm0.1015.06±0.2415.06\pm0.2415.08±0.4415.08\pm0.44
site-success82.93±5.5382.93\pm5.5390.37±0.7890.37\pm0.7886.94±5.8486.94\pm5.8491.47±6.2591.47\pm6.2585.53±5.4385.53\pm5.43
Mean44.4047.1746.2947.2346.59

Table A3. Ablations of MoREGT-small (30M parameters) trained from scratch: TEST AUROC (%, ↑\uparrow) for classification and nMAE (%, ↓\downarrow) for regression. Mean ±\pm std over 3 seeds. Bold: best per row.

Signatures of 371 columns and similarity of column names versus expert agreement

Figure 4. Column names steer expert routing, and only after pretraining. (a) t-SNE of the routing signatures of all 371 columns under the trunk pretrained without rel-f1, coloured by modality. Callouts count the blocks (of 10) in which a rel-f1 column and its most similarly named same-modality column elsewhere pick the same top expert. (b) Share of blocks in which a held-out column and a seen column of the same modality pick the same top expert, against the cosine similarity of their names, pooled over the seven LODO trunks (thin lines: folds; band: 95% CI).

Column names steer routing. Since the router reads only the value-free signature, we route every column offline from the weights of each leave-one-database-out trunk and compare held-out columns with same-modality columns in the seen databases. Agreement is measured against chance: the rate any same-modality pair would show if names were ignored, which depends on how concentrated expert use is. Both controls sit at their own chance level near 100%. At initialisation the modality embedding is about 150 times larger than the variation between names, so every column of a modality routes to the same expert. With the trained name projection’s strength but random directions, names barely register: column-name embeddings occupy a narrow subspace (about 37 effective dimensions of 2,560), which random directions mostly miss. After pretraining, names move the signature as much as modality does. Chance falls to 58.5%, and agreement rises with name similarity, from 44% (cosine <0.5< 0.5, below chance) to 84% (≥0.8\ge0.8). For example, the unseen rel-f1 column results.date routes like rel-trial’s outcomes.date in 9 of 10 blocks. Per fold, the most similarly named column shares the top expert 8.9 points more often than a random same-type column (−0.1 at initialisation, +0.5 with random directions), on six of seven folds. The exception, rel-event, has 100 of 122 columns named c_1. . . c_100.

Other feature learning methods. Table 4 compares agent-derived columns against generic Featuretools transformations and evaluates their transfer to the RT backbone. On nine tasks, MoREGT with agent-derived columns outperforms Featuretools on eight, improving mean AUROC from 82.03 to 83.63 and reducing nMAE from 40.61 to 38.35. The largest differences are on user-attendance (8.81 nMAE) and site-success (5.16 nMAE), while on driver-dnf and study-adverse the gap is within one standard deviation. The exception is driver-position, where Featuretools reaches 32.56 nMAE, the lowest error of any method on that task. Applying the same agent-derived columns to RT yields substantially worse results (76.16 AUROC, 49.70 nMAE), even under the same fine-tuning recipe. These results suggest that the utility of derived columns can vary across relational architectures and fine-tuning configurations.

DatasetTaskMoREGT + agentMoREGT + FeaturetoolsRT + agent
f1driver-dnf ↑\uparrow84.11±\pm0.3983.96±\pm0.7682.93
f1driver-top3 ↑\uparrow92.06±\pm0.7691.12±\pm0.7585.52
eventuser-ignore ↑\uparrow86.31±\pm3.3884.26±\pm1.3471.88
eventuser-repeat ↑\uparrow82.63±\pm2.4280.94±\pm0.7273.56
trialstudy-outcome ↑\uparrow73.05±\pm1.7269.89±\pm0.5066.89
f1driver-position ↓\downarrow37.70±\pm1.9832.56±\pm1.0539.76
eventuser-attendance ↓\downarrow31.09±\pm0.6239.90±\pm1.9056.42
trialstudy-adverse ↓\downarrow9.75±\pm0.279.97±\pm0.8814.67
trialsite-success ↓\downarrow74.84±\pm5.5680.00±\pm7.3187.95
Mean AUROC ↑\uparrow83.6382.0376.16
Mean nMAE ↓\downarrow38.3540.6149.70

Table 4. Agent-derived columns versus generic feature synthesis and the RT backbone. The MoREGT variants share the same LODO pretrained trunk; RT uses its own LODO trunk. All methods use the same fine-tuning recipe. TEST AUROC (↑\uparrow) and nMAE (↓\downarrow), in %. MoREGT results are mean ±\pm std over 3 seeds.

Conclusion

We presented MoREGT, a graph transformer with row and cell tokens, time- and role-aware attention and value-free expert routing, paired with a label-blind agent that adds validated single-row columns to the target database. Under the same LODO pretraining as RT, MoREGT improves on RT, and with derived columns it reaches the highest mean AUROC (79.63) and lowest nMAE (26.98) among the compared methods, despite pretraining on substantially smaller data. After pretraining, columns with similar names share experts across databases. This demonstrates that better architectures and target-specific features thus offer a route to stronger relational foundation for transfer across Web platforms, alongside larger pretraining corpora.

Implementation Details

Table A1 lists the value of every symbol introduced in § 3. All seven leave-one-database-out trunks use the same values.

SymbolValueSymbolValue
Sampling and tokensExperts
Hops, neighbors per edge typeNhop,NnbrN_{\mathrm{hop}}, N_{\mathrm{nbr}}2, 12Cell experts: routed, shared, top-kkNr,Ns,krN_{\mathrm{r}}, N_{\mathrm{s}}, k_{\mathrm{r}}8, 2, 2
Cells per exampleNcellN_{\mathrm{cell}}1024Row experts: routed, sharedNrrow,NsrowN_{\mathrm{r}}^{\mathrm{row}}, N_{\mathrm{s}}^{\mathrm{row}}4, 1
Standardization constantϵσ\epsilon_{\sigma}10−610^{-6}Expert hidden widthdffd_{\mathrm{ff}}1024
Name-embedding dimensiondnd_n2560Parameters (share in experts)—133.9M (95%)
BackboneAgentic loop
Width, blocks, attention headsd,L,−d, L, -256, 10, 8Turn budget, feedback roundsNturn,RN_{\mathrm{turn}}, R200, 3
Rotary frequenciesK,[ωmin⁡,ωmax⁡]K, [\omega_{\min}, \omega_{\max}]8, [0.3, 5]Range fraction, missing-rate slackq,ηq, \eta0.9, 5 points
Signature widthdsd_s128Correlation thresholdrmax⁡r_{\max}0.999
Pooling heads, head widthM,dpM, d_p4, 64
Pretraining objective
Extra seed-row masking ratepmaskp_{\mathrm{mask}}0.5
Loss weightsλcat,λaux,λbal\lambda_{\mathrm{cat}}, \lambda_{\mathrm{aux}}, \lambda_{\mathrm{bal}}4, 0.5, 0.01

Table A1. Hyperparameters of MoREGT and of the agentic loop.

Encoders. Each semantic type has one value encoder, shared by all its columns. A numerical value is standardized to x~i,j=(xi,j−μj)/(σj+ϵσ)\tilde{x}_{i,j} = (x_{i,j} - \mu_j)/(\sigma_j + \epsilon_{\sigma}), where μj\mu_j and σj\sigma_j are the mean and standard deviation of column jj, and encoded as vi,j=wnumx~i,j+bnumv_{i,j} = w_{\mathrm{num}}\tilde{x}_{i,j} + b_{\mathrm{num}}, with missing values mapped to x~i,j=0\tilde{x}_{i,j} = 0. A categorical value cc is encoded through its frozen embedding nj,cn_{j,c}, free text through a frozen sentence encoder, and timestamps through a standardized year and sine–cosine harmonics of calendar fields. Column names, table names (as the string “table TT”), category values and FK roles are embedded once, offline, with Qwen3-Embedding-4B, and free-text cell values with MiniLM-L12-v2. No language model runs inside the training loop.

Initial row embeddings and signatures. The initial embedding of row ii, which the first block reads, is a bias-free projection of the mean of its cells’ initial embeddings together with its signature Ui(0)=Wu[Hˉi(0);Si]U_i^{(0)} = W_u[\bar{H}_i^{(0)}; S_i], where Hˉi(0)\bar{H}_i^{(0)} is the mean of Hi,j(0)H_{i,j}^{(0)} over the cells of row ii that survive the cap NcellN_{\mathrm{cell}}. For a row whose cells were all removed by the cap, Hˉi(0)=0\bar{H}_i^{(0)} = 0, so its initial embedding depends only on its signature. The projections WzW_z, WτW_{\tau}, WTW_T, WρW_{\rho} and Wτ′W'_{\tau} of Eqs. (6) and (7) are bias-free linear maps, and EstE_{\mathrm{st}} and EhopE_{\mathrm{hop}} are lookup tables. The role ρ0(i)\rho_0(i) is found by a breadth-first search over the sampled row graph from the seed, with ties broken by the lowest-indexed frontier predecessor. Roles are direction-folded, so rows reached through the same FK from the parent side or from the child side share one role. The seed row takes a reserved role whose embedding is zero, so its signature is the RMSNorm of its table-name, hop-0 and age terms.

Time in attention. Ages are computed in float64 with Δi\Delta_i in seconds on every database. In each head of width dhd_h, half of the dimensions are rotated, as K=dh/4K = d_h/4 planar pairs with one frequency each, and the other half are left unrotated. The rotation angle of row ii is ωfτi\omega_f \tau_i for a timed row and 0 for an untimed row. An unrotated row would therefore look as recent as a row with Δi=0\Delta_i = 0, which the untimed offset corrects. The clamped indicator of row ii is computed from the raw times as ti>t∗t_i > t^* and is set only for timed rows; it is also the last entry of φi\varphi_i. Two learned scalars, buntimedb_{\mathrm{untimed}} and bclampedb_{\mathrm{clamped}}, both initialized to zero, are added to the attention score of a pair when the query row or the key row is untimed or clamped, respectively. Both are shared by all heads and by cell and row attention. CellAttn receives only a padding mask, the log-ages and these two indicators, so it has no structural mask and no row identifier.

Pooling. Pool is cross-attention with MM heads of width dpd_p from each row to its own cells. Head mm weights the cells of row ii by αi,jm\alpha_{i,j}^{m}, and the heads’ outputs are concatenated into the update that the third line of Eq. (3) adds to the row embedding:

αi,jm=softmax⁡j∈J(i)(WmQ[Ui;Si])⊤WKnjdp,\alpha^{m}_{i,j}=\operatorname{softmax}_{j\in\mathcal{J}(i)}\frac{\left(W^{Q}_{m}[U_i;S_i]\right)^{\top}W^{K}n_j}{\sqrt{d_p}},
Pool⁡i=WO∥m=1M∑j∈J(i)αi,jmWVHi,j′′,(9)\operatorname{Pool}_i=W^{O}\mathop{\|}_{m=1}^{M}\sum_{j\in\mathcal{J}(i)}\alpha^{m}_{i,j}W^{V}H^{\prime\prime}_{i,j}, \tag*{(9)}

where Ui=Ui(ℓ−1)U_i=U_i^{(\ell-1)}, J(i)\mathcal{J}(i) is the set of columns of row ii whose cells survive the cap, WmQW^{Q}_{m} projects [Ui;Si][U_i;S_i] to width dpd_p, WKW^{K} and WVW^{V} project column-name embeddings and cell embeddings to width dpd_p and are shared by all heads, ∥\| concatenates the MM head outputs, and WO∈Rd×MdpW^{O}\in\mathbb{R}^{d\times Md_p}. All four projections are bias-free. A row whose cells were all removed by the cap receives Pool⁡i=0\operatorname{Pool}_i=0.

Experts and routing regularizers. Every expert, routed or shared, is a bias-free SwiGLU network E(H^i,j)=W3(SiLU⁡(W1H^i,j)⊙W2H^i,j)E(\hat{H}_{i,j})=W_3\left(\operatorname{SiLU}(W_1\hat{H}_{i,j})\odot W_2\hat{H}_{i,j}\right) of hidden width dffd_{\mathrm{ff}}. The cell router WgW_g of Eq. (8) is a linear map on Zi,jZ_{i,j}. The row router is a cosine router (Chi et al., [2]) on the row signature, which keeps krk_r of the NrrowN_r^{\mathrm{row}} routed row experts with gates

girow=softmax⁡(TopK⁡kr([cos⁡(Si,werow)/κ]e=1Nrrow)),(10)\mathbf{g}^{\mathrm{row}}_i=\operatorname{softmax}\left(\operatorname{TopK}_{k_r}\left(\left[\cos(S_i,w^{\mathrm{row}}_e)/\kappa\right]_{e=1}^{N_r^{\mathrm{row}}}\right)\right), \tag*{(10)}

where weroww^{\mathrm{row}}_e is the router row of row expert ee, cos⁡\cos is cosine similarity, and κ=exp⁡(ν)\kappa=\exp(\nu) is a temperature with ν\nu a learned scalar initialized at zero. Every row also passes through the NsrowN_s^{\mathrm{row}} shared row experts.

Column types are long-tailed, and since all cells of a type share a route, a uniform load-balancing loss (Fedus et al., [6]) could equalize load only by merging types. We instead keep the routing directions of the cell router distinct without constraining usage:

Ωcell=∥W^gW^g⊤−I∥F2=∑e≠e′cos⁡2(we,we′),(11)\Omega_{\mathrm{cell}}=\left\|\hat{W}_g\hat{W}_g^{\top}-I\right\|_{F}^{2}=\sum_{e\ne e'}\cos^{2}(w_e,w_{e'}), \tag*{(11)}

where wew_e is row ee of WgW_g, W^g\hat{W}_g is WgW_g with ℓ2\ell_2-normalized rows, II is the Nr×NrN_r\times N_r identity, and the sum runs over ordered pairs of distinct routed experts. Ωrow\Omega_{\mathrm{row}} is the same penalty on the row-router rows weroww^{\mathrm{row}}_e. The row router also receives a Switch-style load-balancing term (Fedus et al., [6]), computed over the rows that are not padding:

Brow=Nrrow∑e=1Nrrowfegˉe,(12)B_{\mathrm{row}}=N_r^{\mathrm{row}}\sum_{e=1}^{N_r^{\mathrm{row}}}f_e\bar{g}_e, \tag*{(12)}

where fef_e is the fraction of these rows whose highest-scoring expert is ee, and gˉe\bar{g}_e is the mean over these rows of the gate of ee in Eq. (10), which is zero for rows that do not keep ee. The routing term of the loss sums both levels over the blocks,

Laux=∑ℓ=1L(Ωcell(ℓ)+λauxΩrow(ℓ)+λbalBrow(ℓ)),(13)\mathcal{L}_{\mathrm{aux}}=\sum_{\ell=1}^{L}\left(\Omega_{\mathrm{cell}}^{(\ell)}+\lambda_{\mathrm{aux}}\Omega_{\mathrm{row}}^{(\ell)}+\lambda_{\mathrm{bal}}B_{\mathrm{row}}^{(\ell)}\right), \tag*{(13)}

where the superscript (ℓ)(\ell) marks the terms of block ℓ\ell. Pretraining and fine-tuning both add λauxLaux\lambda_{\mathrm{aux}}\mathcal{L}_{\mathrm{aux}} to the task loss, so the three terms carry effective weights λaux\lambda_{\mathrm{aux}}}, λaux2\lambda_{\mathrm{aux}}^{2} and λauxλbal\lambda_{\mathrm{aux}}\lambda_{\mathrm{bal}}, that is, 0.5, 0.25 and 0.005.

Pretraining. We train each trunk with AdamW (β=(0.9,0.95)\beta=(0.9,0.95), learning rate 3×10−43\times10^{-4}, weight decay 0.1 on matrices only), with 5% warmup followed by linear decay over a fixed budget of 18 hours, gradient clipping at 1.0 and bfloat16 precision. The batch is 64 examples per GPU on 4 H100 GPUs, which gives 208k–237k steps per trunk. Each step draws one seed table, with probability proportional to its number of maskable columns times its number of eligible rows, normalized so that each pretraining database receives an equal share of steps. Pretraining seeds lie before the RelBench validation timestamp, and the last 10% of that window is held out to select the checkpoint.

Fine-tuning and model selection. Fine-tuning uses AdamW with a constant learning rate of 3×10−43\times10^{-4}, weight decay 0.01 and batch size 128 (64 on rel-event, for memory), for at most 12 epochs of 200 batches, with early stopping after 4 epochs without improvement. We report the epoch with the best RelBench validation metric, averaged over three seeds.

Additional Results

Table Table A2 lists the columns the agent added for MoREGT (with agent). Over the 21 tasks it proposed 230 columns, and screening kept 210, between 8 and 14 per task, for $28.59 of API cost in total. All proposals were kept on 17 of the 21 tasks; the 20 rejections are on hm user-churn (1) and the three trial tasks (19). The example columns in the table are of three kinds: indicators that group codes or thresholds (results.mechanical_dnf, posts.is_answer), products of a flag with another flag or an amount (review.verified_complaint, transactions.online_revenue), and differences, ratios and logarithms (results.positions_gained, studies.log_enrollment_per_arm). Compared with MoREGT (w/o agent), the derived columns raise AUROC by 4.12 points on trial study-outcome, 3.10 on avito user-clicks and 2.69 on hm user-churn, and lower nMAE by 3.46 points on trial study-adverse and 2.80 on site-success (Table 1 and Table 2). On the f1 tasks and on event user-repeat, they change scores by at most 0.21 points.

DatasetTaskAcc./prop.Example column
amazonitem-churn9/9review.verified_complaint = (rating <= 2) * verified
item-ltv11/11product.charm_priced = (price % 1) >= 0.9
user-churn9/9review.verified_disappointment = (5 - rating) * verified
user-ltv9/9product.impulse_priced_item = (price > 0) * (price < 5)
avitoad-ctr12/12SearchStream.position_discounted_prior = HistCTR / log1p(Position)
user-clicks9/9SearchStream.paid_top_slot_impression = (ObjectType == 3) * (Position <= 1)
user-visits9/9SearchInfo.search_filters_applied = (CategoryID > 0) + (LocationID > 0)
eventuser-attendance9/9event_interest.invitation_uptake = invited * interested
user-ignore9/9event_interest.ignored_invitation = invited * (1 - interested) * (1 - not_interested)
user-repeat9/9event_interest.organic_interest = interested * (1 - invited)
f1driver-dnf14/14results.mechanical_dnf = (statusId > 4) * (statusId < 11)
driver-position9/9results.positions_gained = grid - positionOrder
driver-top312/12results.front_row_start = (grid >= 1) * (grid <= 3)
hmitem-sales9/9transactions.online_revenue = price * (sales_channel_id - 1)
user-churn9/10customer.subscribed_not_active = (FN > 0) * (1.0 - (Active > 0) * 1.0)
stackpost-votes12/12votes.score_vote_sign = 1 * (VoteTypeId == 2) - 1 * (VoteTypeId == 3)
user-badge12/12badges.achievement_badge_prestige = (TagBased == 0) * (4 - Class)
user-engagement11/11posts.is_answer = 1 * (PostTypeId == 2)
trialsite-success11/19studies.is_underpowered_comparative = (number_of_arms >= 2) * (enrollment < 30 * number_of_arms)
study-adverse8/11studies.log_enrollment_per_arm = log1p(enrollment / maximum(number_of_arms, 1))
study-outcome8/16outcome_analyses.effect_signal_fraction = abs(param_value) / (abs(param_value) + abs(dispersion_value) + 0.000001)

Table A2. Columns derived by the agent for MoREGT (with agent), with one example per task: 210 of 230 proposed columns kept after screening (Acc./prop.), $28.59 of API cost in total. The full list, with every column’s expression, units and rationale, is in the repository (results/derive/accepted_columns.md).

Inference cost. Let a seed’s context hold SS cells in RR rows, and let the model have LL blocks of width dd and FFN width dffd_{\mathrm{ff}}. One MoREGT block costs

O(S2d)+O(R2d)+O((k+ns)Sddff+(kr+nr)Rddff)+O(Sd2),(14)O(S^{2}d) + O(R^{2}d) + O((k+n_s)S d d_{\mathrm{ff}} + (k_r+n_r)R d d_{\mathrm{ff}}) + O(Sd^{2}), \tag*{(14)}

where the last term covers the projections, row pooling and broadcast. Experts are dispatched sparsely: each cell passes through its k=2k = 2 routed and ns=2n_s = 2 shared experts only, so the FFN term does not grow with the number of experts (M=8M = 8). Row attention costs (R/S)2≈3%(R/S)^2 \approx3\% of cell attention. The forward pass is therefore Θ(LS2d)\Theta(LS^2d), the same order as RT, whose every layer runs a full attention over all SS cells next to its column, feature and neighbour attentions. RT-J drops the full attention and sorts cells by column, so its column attention is block-diagonal and its block-sparse kernel skips most blocks, but its worst case is still O(S2d)O(S^2d). Both RT models also materialise S×SS \times S boolean masks, including an S×S×fS \times S \times f foreign-key comparison for ff foreign keys per cell; we build only a padding block mask with (S/128)2(S/128)^2 entries. At our width (d=256d = 256, dff=1024d_{\mathrm{ff}} = 1024, four active experts per cell), the linear FFN term in (14) exceeds the quadratic attention term up to S≈6,000S \approx6{,}000 cells, so at deployed sizes our cost per seed grows almost linearly in SS. Building the context is linear in its size for our temporal neighbour sampler and for RT’s breadth-first sampler. RT-J additionally runs W=104W = 10^4 random walks of length ℓ=20\ell= 20 per seed to find in-context examples, at O(S+Wℓ)O(S + W\ell) per seed whatever SS is. As shown in Figure A1, MoREGT is fastest for inferring 100 rel-f1 driver-dnf test seeds. MoREGT is about 36x faster than RT-J while achieving comparable performance.

Figure A1. Inference time. 100 rel-f1 driver-dnf test seeds.

Each variant in Table A3 removes one component: Dense FFN replaces every mixture of experts with a dense feed-forward layer, Hidden routing routes on the hidden state instead of the value-free signature, No time attn. removes the log-age terms from attention, and No role bias removes the FK-role bias of Eq. (5). Removing time from attention gives the largest drop in both means (1.93 AUROC and 2.83 nMAE), followed in classification by removing the role bias (1.75 AUROC); in regression, Dense FFN is close behind (2.77 nMAE), and both regression gaps come mostly from site-success. The routing ablations matter most on driver-dnf, where Dense FFN and Hidden routing lose 2.30 and 2.46 AUROC with standard deviations below 0.6, against 0.65 for No time attn. On study-outcome, the attention components matter more than routing: removing the time terms or the role bias lowers AUROC by 1.97 and 2.33, against 0.39 and 1.02 for the routing variants, although these gaps are only one to two standard deviations. On rel-event, no variant differs from the full model by more than one standard deviation.

References

  1. [1]Tianlang Chen, Charilaos Kanatsoulis, and Jure Leskovec. 2025. RelGNN: Composite Message Passing for Relational Deep Learning. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=XXh3ZmW2Uyarxiv.org/abs/2502.06784
  2. [2]Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. 2022. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems 35 (2022), 34600–34613.arxiv.org/abs/2204.09179
  3. [3]Damai Dai, Chengqi Deng, Chenggang Zhao, R.X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 1280–1297. doi:10.18653/v1/2024.acl-long.70
  4. [4]Zezhong Ding, Jin Li, Xugang Wang, and Xike Xie. 2026. Gaussian Relational Graph Transformer. arXiv preprint arXiv:2605.15575 (2026).arxiv.org/abs/2605.15575
  5. [5]Vijay Prakash Dwivedi, Sri Jaladi, Yangyi Shen, Federico Lopez, Charilaos Kanatsoulis, Rishi Puri, Matthias Fey, and Jure Leskovec. 2026. Relational Graph Transformer. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026. 154348–154369. https://proceedings.iclr.cc/paper_files/paper/2026/file/fa1ef43326cfb8a808c0649708187f2a-Paper-Conference.pdfarxiv.org/abs/2505.10960
  6. [6]William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res. 23, 1, Article 120 (Jan. 2022), 39 pages.arxiv.org/abs/2101.03961
  7. [7]Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. 2024. Position: Relational Deep Learning - Graph Representation Learning on Relational Databases. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 13592–13607. https://proceedings.mlr.press/v235/fey24a.htmlarxiv.org/abs/2312.04615
  8. [8]Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. 2022. Why do tree-based models still outperform deep learning on typical tabular data?. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 37, 14 pages.
  9. [9]Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2022. DEMix Layers: Disentangling Domains for Modular Language Modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 5557–5576. doi:10.18653/v1/2022.naacl-main.407DOI
  10. [10]Sungwon Han, Jinsung Yoon, Sercan Ö. Arik, and Tomas Pfister. 2024. Large language models can automatically engineer features for few-shot tabular learning. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 695, 26 pages.arxiv.org/abs/2404.09491
  11. [11]Noah Hollmann, Samuel Müller, and Frank Hutter. 2023. Large language models for automated data science: introducing CAAFE for context-aware automated feature engineering. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 1938, 23 pages.arxiv.org/abs/2305.03403
  12. [12]Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous Graph Transformer. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Machinery, New York, NY, USA, 2704–2710. doi:10.1145/3366423.3380027DOI
  13. [13]Xingyue Huang, Louis Tichelman, Jinwoo Kim, Krzysztof Olejniczak, and Ismail Ilkan Ceylan. 2026. RelAgent: LLM Agents as Data Scientists for Relational Learning. arXiv:2605.07840 [cs.LG] https://arxiv.org/abs/2605.07840
  14. [14]Valter Hudovernik, Federico López, Vid Kocijan, Akihiro Nitta, Jan Eric Lenssen, Jure Leskovec, and Matthias Fey. 2026. KumoRFM-2: Scaling foundation models for relational learning. arXiv preprint arXiv:2604.12596 (2026).arxiv.org/abs/2604.12596
  15. [15]Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. 2020. Multiplicative Interactions and Where to Find Them. In International Conference on Learning Representations. https://openreview.net/forum?id=rylnK6VtDH
  16. [16]James Max Kanter and Kalyan Veeramachaneni. 2015. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA). 1–10. doi:10.1109/DSAA.2015.7344858DOI
  17. [17]Ryotaro Kawata, Kohsei Matsutani, Yuri Kinoshita, Naoki Nishikawa, and Taiji Suzuki. 2025. Mixture of Experts Probably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 29390–29448. https://proceedings.mlr.press/v267/kawata25a.htmlarxiv.org/abs/2506.01656
  18. [18]Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. 2024. CARTE: pretraining and transfer for tabular learning. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 956, 24 pages.arxiv.org/abs/2402.16785
  19. [19]Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik, Vijay Prakash Dwivedi, Johannes Hoffart, Carlos Guestrin, and Jure Leskovec. 2026. PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=rM9r9qyRSAarxiv.org/abs/2602.04029
  20. [20]Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Charilaos I. Kanatsoulis, Roshan Reddy Upendra, Mahmoud Mohammadi, Joe Meyer, Tom Palczewski, Carlos Guestrin, and Jure Leskovec. 2026a. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=rpPtgMC5s9arxiv.org/abs/2510.06377
  21. [21]Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal, Charilaos Kanatsoulis, Roshan Upendra, Tom Palczewski, Carlos Guestrin, and Jure Leskovec. 2026b. RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions. arXiv preprint arXiv:2610.17867 (2026).arxiv.org/abs/2610.17867
  22. [22]Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaq Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovec. 2024. RelBench: A Benchmark for Deep Learning on Relational Databases. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 21330–21341. doi:10.52202/079017–0672arxiv.org/abs/2407.20060
  23. [23]Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. 2021. Hash layers for large sparse models. In Proceedings of the 35th International Conference on Neural Information Processing Systems (NIPS ’21). Curran Associates Inc., Red Hook, NY, USA, Article 1343, 12 pages.arxiv.org/abs/2106.04426
  24. [24]Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations. https://openreview.net/forum?id=B1ckMDqlgarxiv.org/abs/1701.06538
  25. [25]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomput. 568, C (Feb. 2024), 12 pages. doi:10.1016/j.neucom.2023.127063DOI
  26. [26]Yanbo Wang, Xiyuan Wang, Quan Gan, Minjie Wang, Qibin Yang, David Wipf, and Muhan Zhang. 2025. Griffin: Towards a Graph-Centric Relational Database Foundation Model. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=TxeCxVb3cLarxiv.org/abs/2505.05568

Paper details

vlnc:W_yye97c5s
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents