Advanced Topics in GNNs — In-Context Learning and Relational Foundation Models

Slide 3 names the two papers of this lecture. PRODIGY: Enabling In-context Learning Over Graphs (Huang, Ren, Chen, Krvzmanc, Zeng, Liang, Leskovec; NeurIPS 2023) and the Relational Transformer (Ranjan, Hudovernik, Znidar, Kanatsoulis): "Toward Zero-Shot Foundation Models for Relational Data". Both ask the same question: can one pretrained model solve a new task on new data without being retrained?

Slides 5–7 build one figure that compares three ways to do machine learning. Each column is a network (a trapezoid) with an input at the bottom and an output at the top.

• Training (slide 5): three separate networks for three tasks. The same input "hello" gives "5" (count letters), "h" (first letter) or "bonjour" (translate), each from its own model.
• Fine-tuning (slide 6): one big shared network (blue) at the bottom, with a small task-specific head on top for each task. You pretrain once, then update weights for each task.
• In-context learning (slide 7): one big network with no task heads. The input itself carries the task: "Thanks → merci, Wall → mur, hello → ?". The model answers "bonjour". This is few-shot prompting.

Slide 8 defines in-context learning (ICL): performing a new task by "learning" from the input context, without any gradient update. The slide calls it very powerful, because now one foundation model can directly answer all tasks. The three small panels show GPT-style prompts for predicting a sum ("3 + 8 = 11, …"), unscrambling words and translating; in each, a few examples come first and the model completes the last line.

Slide 9 asks the lecture's question: what is in-context learning over graphs? Slide 10 gives the plan. PRODIGY is an in-context learner for graphs that solves novel tasks on novel graphs. It has three parts:

1. Prompt Graph representation: put the few-shot prompt for any graph task into one input format that a single shared model can read.
2. Prompt Graph inference: an "in-context prediction GNN" that reads this format.
3. Pretraining: generate many diverse pretraining tasks in the Prompt Graph format (Neighbor Matching and MultiTask).

Slide 12 recalls the three classic graph learning tasks: node classification, link prediction and graph classification. The pictures are a social network, a knowledge graph and a set of small molecules.

Slide 14 shows what ICL means on a graph, using link classification. On the left is text prompting again (different tasks, same model). On the right:

• A graph \(\mathcal{G}\) (a new graph the model has never seen). Some nodes are drawn as coloured circles (head input nodes) and coloured triangles (tail input nodes).
• Prompt examples \(\mathcal{S}\): four pairs (circle, triangle) as input \(x\), each with a label \(y\) drawn as an orange or blue diamond (for example, the relation type of the edge).
• Queries \(\mathcal{Q}\): a new pair (blue circle, blue triangle) whose label is "orange or blue?".

The yellow box states the requirement: an in-context learner for graphs should solve novel tasks on novel graphs without gradient updates. "Different graphs" and "different tasks" are both marked in red: both can change between pretraining and use.

Slides 15–16 state the two challenges and their answers:

1. Format. How do we represent the few-shot prompt for different graph tasks in the same input format, so one shared model can consume it? Answer: the Prompt Graph, which represents each few-shot prompt as a meta hierarchical graph.
2. Pretraining. How do we pretrain a model that can solve any task in this format? Answer: PG pretraining: pretrain a message-passing model on self-supervised tasks written in the Prompt Graph format, over diverse underlying graphs.

Slide 18 shows the full Prompt Graph, "a unified representation of few-shot prompts over graphs for diverse tasks". It has two layers:

• Bottom (orange band), data graphs \(\mathcal{G}^d\): one small circled subgraph per prompt example (four of them, the prompt examples \(\mathcal{S}\)) and one per query (\(\mathcal{Q}\)).
• Top (blue band), the task graph: one square data node \(v_x\) above each data graph, and two diamond label nodes \(v_y\). Edges between them are marked T (green) or F (red), and edges to the query are marked "?".

Slides 19–22 explain step 1, the data graph: it contextualizes each input \(x\) in the graph \(\mathcal{G}\), for example by subgraph extraction. A single node or edge means little on its own; its neighbourhood carries the information.

• Link prediction (slides 19–20): for each input pair (head, tail), extract the subgraph around both endpoints. The figure shows each prompt example and query of slide 14 turned into one circled subgraph containing its circle and triangle.
• Node classification (slide 21): the input is a single node; extract the subgraph around it.
• Graph classification (slide 22): the input is already a small graph, so the data graph is that graph itself.

Slide 23 sums up how the data graph unifies the input format:

• Use a different input node set for classification at different levels: one node for node classification, two nodes for edge classification, all nodes for graph classification. The rest of the pipeline stays the same.
• Use text features to unify node features across datasets (each node's text is embedded by a language model, so all graphs share one feature space).

Slide 24 introduces step 2, the task graph: it interconnects inputs and labels across examples to form context for the queries. Slides 25–31 build it for link prediction, one piece at a time:

1. Slide 26: put one data node \(v_x\) above each data graph (it will hold the embedding of that subgraph), and one label node \(v_y\) for each class (two diamonds here).
2. Slides 27–28, prompt examples: add bidirectional edges between each example's data node and all label nodes. The edge to the true label carries the feature T; the edges to the other labels carry F. So each example tells the label nodes "I am an instance of you" or "I am not".
3. Slides 29–30, queries: add single directed edges from each label node to each query data node (the "?" edges). Information flows from labels to the query, but the query cannot change the label nodes.

Slide 31 shows the finished task graph above the data graphs. Slides 32 and 33 show that the same construction works for node classification (input = single node) and graph classification (input = small graph): only the data graphs change.

Slide 34, flexibility of the task graph. It unifies the classification format:

• A different number of classes just means a different number of label nodes.
• A different number of prompt examples (shots) and queries just means a different number of data nodes and different connections to the label nodes.

The two small pictures are a 2-shot prompt for 2-class classification and a 1-shot prompt for 3-class classification.

Slide 35 explains why this works. On the left, ICL over a graph (graph, prompt examples, query). On the right, the Prompt Graph. The task graph reflects what the task is through the examples (their commonalities and differences), and the data graphs capture all relevant information about each input. The key reframing: ICL over a graph becomes inductive link prediction over a hierarchical graph: predict which label node the query's data node should link to.

Slide 37 shows the inference picture: the whole Prompt Graph goes into a model, which outputs scores for the queries. Slides 38–41 open the model. It does hierarchical message passing over the Prompt Graph, in three stages:

1. Data graph encoder (slide 39). A GNN (the same weights for all, drawn as five blue "GNN" boxes) runs on each data graph and pools the embeddings of the input nodes into the embedding of the data node \(v_x\). This turns "a subgraph around an input" into one vector.
2. Message passing over the task graph (slide 40). A second GNN, a GNN with attention, runs on the task graph. Label nodes collect messages from the example data nodes (with T/F edge features), so each label node becomes a summary of "what my examples look like". Query data nodes receive messages from the label nodes.
3. Compute logits (slide 41). For each query and each label, \[ \text{logit} = \text{cosine\_similarity}(h_{\text{label}},\, h_{\text{data}}). \] The predicted class is the label node with the highest score.

Intuition: the label node embeddings play the role of class prototypes, but they are not fixed. They are built at inference time from the examples in the prompt, so a new set of classes needs no new weights. Check yourself: if you give a 5-way prompt to a model pretrained only on 3-way prompts, what changes in the computation? (Only the number of label nodes; the GNN weights are reused.)

Slide 43: to pretrain, we generate pretraining tasks in the Prompt Graph format and pretrain the model over them. The pipeline is: pretraining data/graph → data generation → Prompt Graph → model → optimize loss. Slide 44 points at the "data gen" step.

Slides 45–47 split data generation into two stages:

1. Few-shot prompt generation from the pretraining graph, by one of two routes:
   • MultiTask (slide 46): simply select several labelled tasks that exist on the pretraining graph and sample few-shot prompts from them. This is supervised pretraining.
   • Neighbor Matching (slide 47): a self-supervised task that needs no labels.
2. Convert to Prompt Graph, with augmentation (for example, dropping nodes or corrupting features in the data graphs).

Slide 48, Neighbor Matching. Idea: classify which neighbourhood a node is in, where each neighbourhood is defined by other nodes in it. The figure shows one large graph with four sampled local neighbourhoods \(\mathcal{G}^d_1, \dots, \mathcal{G}^d_4\) (dotted circles, each around a coloured centre node). Each neighbourhood becomes a class. Prompt examples: four coloured centre nodes, each labelled with its neighbourhood's diamond (two orange, two blue). Queries: two yellow nodes; the "?" arrows ask whether a yellow node lies in the orange or the blue neighbourhood (the dashed circles show 2-hop neighbourhoods). The labels come for free from the graph structure, so we can make unlimited tasks of any number of ways.

Slide 49: mixing different data generation pipelines (Neighbor Matching + MultiTask) gives more diverse pretraining data and better results.

Slide 50, the pretraining objectives:

• a classification loss over the generated tasks (pick the right label node for each query);
• an attribute prediction loss: reconstruct the features that were corrupted during data-graph augmentation.

Why this works: the model never sees the downstream task, but it practises the skill of reading a prompt and matching a query to the examples. That skill is what transfers.

Slide 51, evaluation and datasets. Two pretraining graphs: MAG240M, a large-scale citation network, and Wiki, a knowledge graph built from Wikipedia (Xiong et al., 2018b). After pretraining, in-context performance is measured on different downstream graphs: paper category prediction on arXiv, and KG link prediction on FB15K-237, NELL and ConceptNet.

Slide 52, baselines:

• NoPretrain: same architecture, random initialization.
• Contrastive: standard contrastive pretraining with the same augmentation, then a hard-coded nearest-neighbour rule to fit the in-context setting.
• Finetune: a linear classification head trained on top of the contrastively pretrained encoder (so it does use gradient updates).

And three variants of PRODIGY: PG-NM (Neighbor Matching only), PG-MT (MultiTask only), and PG-ALL (both; shown as "PRODIGY" in the table).

Slide 53: graphs have different features. PRODIGY handles this with text embeddings from a pretrained language model (for example BERT).

Slides 54–57 show one table: pretrain on MAG, then in-context classification of arXiv paper categories. Accuracy (%) by number of classes (ways):

• Result 1 (slide 55): PRODIGY improves performance in all cases, up to 48% over Contrastive and 3× over Finetune (40 ways: 25.13 vs 17.02 and 8.04). Notice that NoPretrain is at chance level (about 100/ways).
• Result 2 (slide 56): PG-NM alone is already strong. Its pretraining had no idea about classifying paper categories, yet it induces strong in-context learning: very different self-supervised tasks are enough.
• Result 3 (slide 57): mixing pipelines (PRODIGY = PG-ALL) gives the best numbers.

Why does Finetune lose so badly with many classes? With 40 classes and only a few shots per class, a newly trained linear head overfits; PRODIGY builds class prototypes from the prompt and needs no training.

Slide 58, scaling with more shots. The plot is ConceptNet, 4-way, accuracy against the number of shots (1 to 20). PG-ALL (orange) rises from about 0.43 to about 0.63; Contrastive (blue) rises from about 0.38 to about 0.51 and flattens early. The message: the model was pretrained with only 3 shots, but it can learn from much more than 3 shots in the context. It generalizes to longer prompts than it saw.

Slide 59, data scaling. Accuracy against the number of training steps (log scale, \(10^3\) to about \(2\times10^4\)). PG-ALL climbs steadily from about 0.455 to about 0.60; Contrastive is noisy and stays around 0.50–0.53. The message: hard and diverse pretraining-task generation lets the model keep scaling with more training data.

Slide 60, conclusion. PRODIGY enables in-context learning over graphs with a new task representation, the Prompt Graph, plus a matching pretraining architecture and objectives:

• The Prompt Graph is a unified representation of few-shot prompts over graphs for diverse tasks.
• Pretraining a hierarchical message-passing model over Prompt Graphs enables in-context learning on unseen tasks on unseen graphs.
• Hard and diverse pretraining tasks let the model keep improving with more training data.

The second part (by Charilaos Kanatsoulis) is about transformers for relational data, towards foundation models for forecasting over relational databases. It builds on Lecture 13.

Slide 62 recalls Relational Deep Learning (RDL; Fey et al. 2023) as a four-step pipeline: relational DB → graph problem → graph ML → solutions.

Slide 63, the key insight: a relational database is an entity-level graph. Every entity (row) is a node, and edges are defined by foreign-key–primary-key links. The figure turns three tables (Users, Sales, Products) into a graph: user icons on the left, sales amounts in the middle, product icons on the right, with coloured bands showing which user bought which product.

Slide 64: predictive tasks become graph ML problems.

• Node-level: churn, lifetime value.
• Link-level: product affinity, recommendation.
• Graph-level: fraud detection.

Slide 65 lists RDL architectures: RDL-GNN (Robinson et al. 2024), RelGNN (Cheng, Kanatsoulis, Leskovec 2025), the Relational Graph Transformer RelGT (Dwivedi et al. 2025) and RelLLM (Wu et al. 2025).

Slide 66, the disadvantages of end-to-end RDL. The figure adds churn labels ("Churn"/"Active" per month) to the users. Two problems:

1. Schema- and task-specific: the model is built and trained for one database and one task.
2. Limited transfer: it cannot generalize to new databases or new tasks.

Slide 67, the goal: bring the success of foundation models (LLMs, ViT) to relational data. The pictures are an LLM and the Vision Transformer, which splits an image into patches and feeds them as tokens to a transformer encoder.

Slide 68, the promise of a relational foundation model:

• Zero-shot forecasting on new databases and unseen tasks.
• Few-shot / fine-tuning with limited supervision.
• Broader impact: democratize AI for structured data. Non-experts get accessible predictive tools; experts get a strong pretrained initialization.

Slide 69, the Relational Transformer (RT). Key idea: treat each database cell as a token and model the relationships between cells with Relational Attention. Three core innovations: (1) cell-level tokenization, (2) task-table integration, (3) relational attention. The slide's figure has three panels:

• (a) Schema: tables User (UserID (P), Gender), Sale (Time, UserID (F), ItemID (F), Stars), Item (ItemID (P), Price, Details), plus a Task table (Time, ItemID (F), Label) for the task "predict total sales over next week". (P) = primary key, (F) = foreign key.
• (b) Context window: a set of cells taken from these tables. Task rows from the past provide in-context labels; the target row has its label cell masked ("?").
• (c) Architecture: for all cells, a per-datatype encoder (num | text | bool | time) plus a frozen language-model embedding of "<column> of <table>"; then \(N\) Relational Transformer blocks, each with Column, Feature and Neighbor attention, then Full attention, then an MLP; then a per-datatype decoder that outputs the predicted cell value.

Slide 70 shows the running example: three linked tables, Transactions, Products and Users; lines show foreign-key links from transaction rows to product rows and user rows.

Slide 71: prior work (RDL-GNN, RelGNN, RelGT, RelLLM) tokenizes at the row level: one row = one node/token, with a table-specific encoder for its columns. This leads to schema-specific architectures: a new database with different columns needs new encoders.

Slide 72, RT's cell-level tokenization. Each token is a triple \[ \text{token} = (\text{value},\ \text{column},\ \text{table}). \] The table and column are given as text descriptions embedded with MiniLMv2 (a small language model). Benefits:

• It unifies relational databases of any schema into one consistent input space: every database is just a set of (value, column-name, table-name) tokens.
• It leverages cell-to-cell dependencies: attention can relate single cells, not only whole rows.

Slide 73, task label integration. The task itself becomes a table, here User-Churn with columns Fkey (the user), Val (the label) and Time. Its rows link to Users like any other table. Benefits:

• Any node-level task becomes masked token prediction (MTP): hide the Val cell of the target row and predict it.
• It adds autoregressive labels as task tokens: past rows of the task table, with their labels visible, sit in the context as in-context examples.

Compare with PRODIGY: there, labels were label nodes in a task graph; here, labels are ordinary cells in a task table. Both put the examples into the input instead of into the weights.

A database is too big for one transformer context, so RT samples a context window around each prediction. Slide 74 lists the sampling rules:

1. Sample a seed node (a row of the task table) and hide its value.
2. Always include parent nodes (follow F → P keys: from a foreign key to the row it points to).
3. Subsample child rows (follow P → F keys: rows that point to this row). Slide 74 calls this a breadth-first search over children.
4. Only sample from previous timestamps.

Slides 75–80 animate this on the example:

• Slide 75: the seed is a User-Churn row; its Val cell is hidden (the smiley).
• Slide 76: its Fkey is user 1, so the parent row of user 1 in Users is included.
• Slide 77: from user 1, child rows in Transactions (Fkey = 1) are sampled; one has product Fkey 100.
• Slide 78: that transaction's parent, product 100, is included.
• Slide 79: a second transaction of user 1 (product 98) and its parent product 98 are added.
• Slide 80: another User-Churn row of user 1 with time \(t_1 \lt t_0\) is included; its label is visible, so it acts as an in-context example. Rows after \(t_0\) are never sampled.

Why rule 2 is "always" and rule 3 is "subsample": a row has few parents (one per foreign key) but may have thousands of children (a user's transactions). Rule 4 avoids temporal leakage: the model must not see information from the future of the moment it predicts.

Plain full attention over a set of cell tokens would ignore the structure of the database. RT therefore uses four attention types in each block (slide 81 shows the sampled context with all included cells outlined). The slides highlight, for one query cell (the face emoji), which cells it may attend to:

1. Column attention (slide 82): a cell attends to the cells in the same column of the same table. The query is the hidden Val cell of the target User-Churn row; it sees the Val cells of the other User-Churn rows, that is, the in-context labels. This is how RT learns "what values this column usually takes".
2. Feature attention (slide 83): a cell attends to the other cells of its own row and of its parent rows (F → P). The target User-Churn row sees its own cells and the whole Users row of user 1. This mixes the features of an entity.
3. Neighbor attention (slide 84): a cell attends to the cells of its child rows (rows that point to it through a foreign key). A cell of the Users row sees the user's Transactions rows and User-Churn rows. This is the message-passing step of a GNN, written as masked attention.
4. Full attention (slide 85): every cell attends to all cells in the context, with no mask.

Intuition: column, feature and neighbor attention are relational masks. They put the schema's structure (columns, rows, keys) into the attention pattern as an inductive bias, while full attention keeps the flexibility of a transformer. Since the masks depend only on "same column", "same row" and "key links", they work for any schema.

Slide 86: RelBench, 7 diverse datasets in four areas: e-commerce (rel-amazon, rel-avito, rel-hm), social (rel-event, rel-stack), sports (rel-f1) and medical (rel-trial).

Slides 87–88, the setup:

• Tasks: binary classification and regression.
• Pretraining: leave-one-database-out. Pretrain on the other databases, then test on the unseen schema. This is a true zero-shot test: the model has never seen these tables or columns.
• Baselines: Griffin (Wang et al. 2025), a universal feature encoder followed by a GNN; Gemma LLMs, prompting Gemma3 with a text-serialized database subgraph; RDL-GNN (Robinson et al. 2024), the schema-specific GNN.

Slide 89, RT pretraining:

• Backbone: 12 layers, 22M parameters.
• Pretraining task: masked token prediction on RelBench.
• Training: 50k steps on 8×A100 GPUs.
• A linear decoder.
• Objective: Huber loss for regression, binary cross-entropy (BCE) for classification.

The Huber loss is quadratic for small errors and linear for large ones, so a few extreme target values (common in sales or counts) do not dominate training.

Slide 91, fine-tuning tests: test-set learning curves up to 32k fine-tuning steps (batch size 256), x-axis on a log scale.

• Left, binary classification (mean of 5 tasks, AUROC %): pretrained RT starts at 70.8 zero-shot, pretrained Griffin at 68.5. Untrained models (RT, RDL-GNN, Griffin) start near 55 or lower. Pretrained RT stays on top and reaches about 80.
• Right, regression (mean of 4 tasks, \(R^2\) %): pretrained RT starts at 29.0 zero-shot, pretrained Griffin at 17.7. Untrained models start below zero and need about \(10^3\) steps to catch up.

Slide 92, zero-shot classification (AUROC, 10 tasks). Mean AUROC: Gemma 4B 63.3, Gemma 12B 62.4, Gemma 27B 64.6, EntityMean (0 parameters) 66.7, Griffin 64.0, RT 70.1 when the target database is not in pretraining. When it is (but the task label is still unseen), Griffin 67.3 and RT 72.9. Note that bigger Gemma models are not reliably better.

Slide 93, zero-shot regression (\(R^2\) %, 8 tasks; the global-mean baseline scores 0.0). Mean \(R^2\): EntityMean 19.3, Griffin 8.9, RT 22.8 (unseen database); Griffin 10.1, RT 26.4 (seen database). RT is the only neural method to beat EntityMean, the non-neural baseline that predicts an entity's own historical average. LLM baselines are poor here.

Slide 94: RT has 1000× fewer parameters and needs a 100× smaller context, yet gives 13% better zero-shot AUROC than Gemma3-27B.

Slide 95, discussion: RT is capable of robust zero-shot transfer in relational settings and allows rapid fine-tuning when supervision exists; there is strong evidence that relational databases, despite diverse applications, share transferable patterns; it is a step toward foundation models for structured enterprise data.

Slide 96, limitations: no link prediction or recommendation tasks yet; trained on limited data; possible improvements in cell encoders and positional biases.