Advanced Architectures in Relational Deep Learning

The lecture opens with a recap of Lecture 12.

Slide 4: valuable data in commerce, finance and social media is stored in relational databases (three schema diagrams: tables as boxes, key links as lines). Slide 5: typical predictive questions on such data: which products will a user purchase in the next 7 days? Will an active user churn in the next 90 days? What will the total sales of each product be in the next 30 days?

Slide 6, insight: data is a graph. The small schema has Users (ID), Sales (ID, Product_ID, Session_ID, User_ID, Price) and Products (Product_ID). Slide 7 draws the database as a graph: users on the left, sale nodes in the middle with their prices ($20, $80, $15, …), products on the right. Rows become nodes; key references become edges.

Slide 8, ML in the language of graphs:

• node-level: churn, lifetime value, next best action;
• link-level: product affinity, recommendations;
• graph-level: fraud, money laundering.

Slide 9, the pipeline: Relational DB → Graph problem → Graph ML → Solutions.

Slide 10, the training table: a special table of training labels with rows (entity ID, time, label); it supports classification, regression and multi-class tasks. The example table has columns Entity ID, Timestamp, Label (e.g. entity 99 with label 1, entity 100 with label 0).

Slide 11, the four-panel RDL pipeline: (a) relational tables with a training table (Products, Transactions, Customers, and Customer LTV with 90-day LTV, User_ID, SeedTime); (b) entities linked by foreign keys; (c) the relational entity graph; (d) the GNN computation trees for the training rows A and B.

Slide 12, the relational entity graph: create connections via primary–foreign keys.

Slide 13: training labels, with timestamps, are attached to the graph (for example, one user has "May '23 Churn" and "June '23 Active").

Slide 14, a GNN on the entity graph: each node's neighbourhood defines its computation graph, and nodes learn how to use information from neighbours optimally. The same user has different computation graphs for different label times (May '23 vs June '23), because only earlier rows can be used.

Slide 15 asks again: why not use an LLM? Slide 16, "some have tried": the paper "Tackling prediction tasks in relational databases with LLMs" (Wydmuch, Borchmann, Graliński; Snowflake AI Research) builds one LLM example by following the keys from a task-table row to the related Customers, Transactions and Products rows and serialising them into a JSON document. Slide 17 shows such a prompt: a full slide of dense JSON text for one example.

Slide 18 lists the reasons:

• LLMs are not trained to understand and learn from the relational nature of databases.
• Database prediction is not sequence modelling.
• LLMs are not trained to predict the future.
• Context size is limited.

Slide 19, RDL gains over LLM: entity classification, ROC-AUC (%) on the RelBench test set, comparing in-context learning with Llama 3.2 3B against RelGNN:

• rel-amazon: user-churn 62.55 vs 70.99 (13% relative gain); item-churn 73.41 vs 82.64 (13%).
• rel-avito: user-visits 53.36 vs 66.18 (24%); user-clicks 54.07 vs 68.23 (26%).
• rel-event: user-repeat 70.11 vs 79.61 (14%); user-ignore 68.65 vs 86.18 (26%).
• rel-f1: driver-dnf 80.03 vs 75.29 (−6%); driver-top3 87.11 vs 85.69 (−2%).
• rel-hm: user-churn 63.81 vs 70.93 (11%).
• rel-stack: user-engagement 81.23 vs 90.75 (12%); user-badge 79.99 vs 88.98 (11%).
• rel-trial: study-outcome 59.17 vs 71.24 (20%).

RelGNN wins on 11 of 13 tasks, often by double digits. The two exceptions are both rel-f1, a small database (Formula 1 results), where the LLM's world knowledge about drivers may help. The slide ends with "stay tuned for more discussion on LLMs later" (Lecture 16).

Slides 21–23 build one message: relational databases hold multi-modal, multi-scale information.

• Entity level (slide 22): each row mixes modalities: text, image, numerical, categorical, timestamp. The schema picture: Products (ProductID, Description, Image, Size), Transactions (TransactionID, ProductID, Timestamp, CustomerID, Price), Customers (CustomerID, Name).
• Database level (slide 23): there is relational structure (how rows link across tables) and temporal structure (when rows were created).

Slide 24, the structure of relational databases has recognisable patterns:

• Tripartite patterns: users – sales – products. Every user–product relation passes through a middle layer of sale rows.
• Cycle patterns: two users each bought the same two products (a T-shirt and sneakers), giving a cycle user → sale → product → sale → user → sale → product → sale → back.
• Structure changes over time: new sale rows keep arriving, so the graph grows.

The point: a generic GNN or transformer treats all of this the same way. Architectures that know about modalities, the tripartite shape and time should do better. The rest of the lecture gives two such architectures (RelGNN and RelGT) and then a foundation model.

Slides 26–27 compare two graphs of the same purchase data:

• A standard heterogeneous graph: customer —purchase→ product. The edge itself carries the meaning.
• The relational entity graph: customer — transaction — product. The purchase is a row (node) of the Transactions table, linked to both sides by keys.

Slide 28, observation: edges in relational entity graphs are defined by primary–foreign key links that merely record table connectivity, without semantics. An edge says "this row's foreign key points to that row", nothing more.

Slide 29: databases introduce junction tables, which decompose each many-to-many association into a pair of one-to-many links. The figure: on the left, customers 1, 2 and products 1, 2 with crossing many-to-many edges; on the right, the same information stored with transactions 1, 2, 3 in the middle, each pointing to one customer and one product.

Why databases do this: a relational column can hold only one value, so a customer row cannot list all its products. The standard design (normalisation) is a separate table whose rows are the pairs. The cost for graph learning: every customer–product relation becomes a 2-hop path through a node that is often just bookkeeping.

Slide 31 recalls the 7 RelBench datasets: E-commerce (rel-amazon, rel-avito, rel-hm), Social (rel-event, rel-stack), Sports (rel-f1), Medical (rel-trial).

Slides 32–34 show their schema graphs, with pink nodes for entity tables and purple nodes for junction tables:

• rel-amazon: customer — review — product.
• rel-hm: customer — transactions — article.
• rel-F1: entity tables constructors, races, drivers, circuits; junction tables constructor_standings, constructor_results, standings, results, qualifying, each pointing to two or three entity tables.

Slide 34 names two shapes: a bridge structure (a junction table linking two entity tables: standings → races, drivers) and a hub structure (a junction table linking three or more: results → races, constructors, drivers).

Slide 35, the limitations of standard message passing on such graphs:

1. Imbalance: junction nodes are 1-hop neighbours, while the more informative nodes become 2-hop. For a customer, the 1st-hop neighbour is a transaction and the product is only a 2nd-hop neighbour.
2. Redundancy: information from the source node is passed to the junction node in the first hop and routed back to itself in the second hop, where it is aggregated together with the junction and 2-hop node information. The figure: customer → transaction (1st hop), transaction → customer (2nd hop).

Slide 36, atomic routes (RelGNN, Chen, Kanatsoulis, Leskovec, ICML 2025). Informally, an atomic route is a simple path between pink (entity) node types, going through purple (junction) types if needed. They can be derived automatically from the schema, without manual intervention or domain knowledge. On the rel-f1 primary–foreign key graph the examples are: circuits → races (a direct link); races ← standings → drivers (a bridge); races ← results → drivers (one route through the results hub).

Slide 37, composite message passing: aggregate messages along atomic routes in a single step. For a route (src, mid, dst),

The junction node's features and the source node's features are first fused, and the fused message goes straight to the destination. An instantiation:

i.e. an attention-weighted sum (weights \(\alpha\)) of the fused messages plus a projection of the destination's own state. The figure contrasts standard message passing (customer ← transaction ← product takes 2 hops, and the customer's message goes to the transaction and back) with composite message passing (transaction and product are fused into one message that reaches the customer in a single hop).

This fixes both problems of slide 35: the product is now a 1-hop source (no imbalance), and the customer's own state is not echoed back through the junction (no redundancy).

Slide 38, results (Hetero-GNN vs RelGNN):

• rel-F1, predict whether a driver will qualify in the top 3 for a race in the next month: ROC-AUC, 13% improvement.
• rel-trial, predict the success rate of a trial site in the next year: MAE (lower is better), 25% improvement.
• rel-amazon, predict the list of distinct items each customer will purchase and review in detail in the next 3 months: MAP, 11% improvement.

RelGNN achieves state-of-the-art performance on the vast majority of RelBench tasks, with improvements of up to 25%.

Slides 40–41 recall the key components of a Transformer (slide credit Joshua Robinson; the header cites You, Ying, Leskovec, "Design Space of Graph Neural Networks", NeurIPS 2020):

1. tokenizing (the inputs \(x_1,x_2,\dots\));
2. positional encoding (added to each token);
3. self-attention (the Transformer block over all tokens).

Key question: what should these be for a graph input? How do we choose them for graph data?

Slides 42–44, processing graphs with Transformers. A graph Transformer must take as input (1) node features, (2) adjacency information, (3) edge features. There are many ways to do this; different approaches correspond to different "matchings" between the graph inputs (1)–(3) and the Transformer components (1)–(3). Slide 44 draws the standard matching (Lecture 8): node features → tokens, adjacency information → positional encoding, edge features → self-attention (as attention biases).

Slide 45, PEARL + Transformer: how to use PEARL (a learnable positional encoding) in practice:

1. Sample node IDs from a probability distribution (random input vectors \(\mathbf q^{(1)},\dots,\mathbf q^{(M)}\)).
2. Process each set of node samples independently with a GNN, on the graph with node attributes removed (giving \(\mathbf P^{(1)},\dots,\mathbf P^{(M)}\)).
3. Summarise the outputs by an empirical expectation, \(\mathbf P=\rho[\cdot]\) over the \(M\) samples.
4. Concatenate the PEARL embeddings with the node features \(\mathbf X\).
5. Pass through the main GNN/Transformer as usual (output \(\mathbf Y\)).
6. Backpropagate to train PEARL and the prediction model jointly.

Why remove node attributes in step 2: the PE should encode structure only (where a node sits), so the GNN sees the bare graph with random inputs; averaging over samples makes the result stable and equivariant to node relabelling.

Slide 46: in a (text) Transformer, an input token is represented by 2 elements: the token id (its feature) and its position (its structure). The picture is the original encoder–decoder Transformer with positional encodings added at the inputs.

Slide 47: in a graph Transformer, a token (node) is also 2 elements: the node feature (feature) and a graph positional encoding (structure). The picture is a graph Transformer layer.

Slide 48: but most graph Transformers are designed for homogeneous, static graphs. RDL adds heterogeneity and temporality, so the design must be improved.

Slide 49: for RDL, the inputs a graph Transformer must take become: node features, adjacency information, edge features (crossed out: PK–FK edges carry no features), node types, and timestamps.

Slide 50, the Relational Graph Transformer (RelGT). Pipeline figure: relational entity graph → tokens (for each seed node, a set of local-context nodes) → Transformer network (\(L\) layers; the seed attends to its local tokens and to a set of global centroids) → downstream layers → prediction. The key idea: represent graph structure with multiple elements, not one: node feature + <multiple information elements> that together encode structure.

Slide 51, limitations of existing positional encodings for RDL:

• designed for homogeneous graphs;
• no temporal dynamics;
• computational overhead (e.g., Laplacian eigenvectors of a graph with millions of rows);
• the graph changes over time (so a PE computed once becomes stale).

Slide 52, the unique properties of relational entity graphs and the matching fixes. The graph is:

• large-scale,
• heterogeneous → add a categorical encoding of node type,
• temporal → add a time encoding.

Slide 53: for the large scale, we cannot attend over the whole graph. Solution: sample a subgraph around the seed node.

Slide 54: training (seed) nodes are the nodes of the type on which the task is defined (e.g., customers for churn). For each seed node we sample a context (subgraph). The figure maps the entity graph to a "seed node + local context" token set.

Slide 55, token preparation (sampling stage):

• For each training node, a fixed set of \(K\) tokens from its local neighbourhood (e.g., up to 2 hops) is selected with temporal-aware sampling (only rows from before the seed time). With \(B\) training nodes in a batch, this gives \(B\times K\) tokens.
• Structure is encoded with a GNN positional encoding for each node in the context, plus its hop distance from the seed node.

Slide 56: each of the \(K\) context nodes is represented by a 5-tuple:

Compare with slide 46: a text token has 2 elements; a RelGT token has 5. Node type handles heterogeneity, time handles temporality, and hop distance plus the GNN PE handle structure.

The GNN PE here is computed on the small sampled subgraph, not on the full database graph. That keeps it cheap and always current, which answers the "computational overhead" and "graph changes over time" limitations of slide 51.

Slides 57–60 build the encoder picture step by step.

Slide 57: seed nodes and their local tokens. Slide 58: for each local token, the five elements are extracted: node feature, node type, hop distance, time, and the sampled subgraph (for the GNN PE).

Slides 59–60: each element goes to its own element-wise encoder, and their combination becomes the token feature fed to the Transformer:

• node feature → a Torch Frame encoder (handles the row's mixed columns: numerical, categorical, text, timestamps);
• node type → nn.Embedding (a learned vector per table);
• hop distance → nn.Embedding (a learned vector per distance 0, 1, 2, …);
• time → a time encoder (of the time relative to the seed);
• subgraph → a GNN encoder that produces the GNN positional encoding.

Slide 60 also draws the sampled subgraph next to the GNN encoder, showing that the PE is computed from it.

Slide 61, ablation studies (Table 2: relative drop in % after removing one component; negative means the component matters). Averages over the eight tasks:

• no global module: −3.87;
• no GNN PE: −5.95;
• no node type: −1.75;
• no hop distance: −2.19;
• no relative time: −9.91.

Every element helps on average, and relative time helps most. The extreme case is rel-hm item-sales without relative time: −77.24. Some entries are positive (e.g., rel-avito user-clicks gains 7.79 without the global module), so not every component helps on every task. Takeaway: the multi-element tokenization is what lets RelGT capture arbitrary structure in relational data.

Slide 62, results (Table 1, RelBench test set; RDL = HeteroGNN baseline, HGT = heterogeneous graph Transformer, HGT+PE = HGT with Laplacian PEs; relative gain over RDL):

• Entity regression (MAE, lower is better). Examples: rel-trial site-success RDL 0.400 → RelGT 0.3263 (+18.43%); rel-avito ad-ctr 0.041 → 0.0345 (+15.85%); rel-f1 driver-position 4.022 → 3.9170 (+2.61%); rel-stack post-votes 0.065 → 0.0654 (−0.62%).
• Entity classification (AUC, higher is better). Examples: rel-f1 driver-top3 0.7554 → 0.8352 (+10.56%); rel-f1 driver-dnf 0.7262 → 0.7587 (+4.48%); rel-avito user-clicks 0.6590 → 0.6830 (+3.64%); rel-stack user-badge 0.8986 → 0.8632 (−3.94%); rel-hm user-churn 0.6988 → 0.6927 (−0.87%).

Takeaway: Relational Graph Transformers improve over GNNs on most tasks, and clearly over plain HGT and HGT+PE (generic graph Transformers). The gains are not universal: on a few tasks the GNN baseline remains better.

Slide 64, limitations of all previous models (GNNs, RelGNN, RelGT):

• they are schema- and task-specific (trained for one database and one training table);
• they do not transfer knowledge from one database to another or from one task to another;
• they do not generalise to new databases or tasks.

Slide 65, can we do better? Goal: bring the success of foundation models (LLMs, ViT) to relational data.

Slide 66, the Relational Foundation Model (RFM) figure. One model receives questions such as "What will user 123 buy next?", "Will user 321 churn soon?", "Is transaction 557 fraudulent?", "Will clinic 987 have high appointment demand?", "Is patient 234 at risk of hospital readmission?". It produces predictions, explanations and evaluations. It works across schemas: (a) e-commerce (Users, Orders, Likes, Items), (b) finance (Customers, Transactions, Account, Branch), (c) health care (Patient, Record, Visit, Doctor). The caption lists the key capabilities: apply to new, unseen databases with very different structure; handle any predictive task type, from one-off assessments (entity-level fraud) to temporal queries (recommendation); generalise to new tasks without task-specific tuning; and support explanations and quantitative evaluation to build trust.

Slide 67, RelGT as the backbone of the RFM (KumoRFM, "A Foundation Model for In-Context Learning on Relational Data", May 2025). The pipeline:

1. A task is written as a query in a predictive query language, e.g. PREDICT COUNT(orders.*, 0, 7) FOR users.user_id = 0 (how many orders will user 0 place in the next 7 days?). A PQL parser and validator check it.
2. A backward-looking graph sampler builds context examples: subgraphs around entities at earlier times \(\hat t\lt t\), together with their known labels \(y\) (a forward-looking sampler computes those labels from what happened after \(\hat t\)).
3. The prediction subgraph is the queried entity at time \(t\), with unknown label "?".
4. A Relational Graph Transformer (\(L\) times) encodes context and query, and in-context learning produces the prediction (with an "explain" path back).

So RelGT is the backbone of the first relational foundation model with zero-shot capabilities powered by in-context learning (ICL): no training on the new database or task.

Slide 68, time to first prediction on node classification tasks (user-engagement, user-badge, user-churn (amazon), item-churn, driver-dnf, driver-top3, user-churn (hm), study-outcome), log scale. KumoRFM is orders of magnitude faster: about 1 second vs about 30 minutes (RDL) vs about 12.3 hours (data scientist) of human labour, and needs about 1 line of code vs about 56 (RDL) vs about 878 (data scientist).

The trajectory of Lectures 12–13: hand features (hours, ~900 lines) → task-specific RDL (minutes, ~50 lines) → foundation model with in-context learning (seconds, one query). It mirrors the path NLP took from feature engineering to fine-tuned models to prompting.