Towards Foundation Models for Knowledge Graphs

The lecture opens with a tour of foundation models in different fields, to set the goal: can we build one for knowledge graphs?

• Large language models (slide 4). Left: an Economist piece, "How does ChatGPT actually work?", which says LLMs are "a giant exercise in statistics". Right: the GPT-3 paper, "Language Models are Few-Shot Learners" (Brown et al., OpenAI). Its Figure 1.2 plots accuracy against the number of examples in the context (zero-shot, one-shot, few-shot) for models of 1.3B, 13B and 175B parameters. Larger models use in-context examples much more efficiently: the 175B curve climbs steeply, the 1.3B curve stays near zero.
• Computer vision (slide 5): an image of a building with regions labelled by "DALLE 2 + SAM" and "DALLE 2": a generative model (DALL·E 2) combined with a general segmentation model (Segment Anything).
• Relational foundation model (slide 6): one frozen model answers questions over many databases: "What will user 123 buy next?", "Will user 321 churn soon?", "Is transaction 557 fraudulent?", "Will clinic 987 have high appointment demand?", "Is patient 234 at risk of hospital readmission?". The figure shows three schemas, (a) e-commerce, (b) finance, (c) health care, and the model outputs predictions, explanations and evaluation. The caption lists the key capabilities: apply to new, unseen databases and schemas; any predictive task type; new tasks without task-specific tuning; explanations. (This is the topic of Lectures 13–14.)
• Biology (slide 7): a New York Times article, "A.I. Is Learning What It Means to Be Alive", and the architecture of ESM (EvolutionaryScale): protein tracks (sequence, structure tokens, secondary structure SS8, solvent accessibility SASA, function) are embedded and summed, passed through transformer blocks with geometric attention, and projected back to logits for each track.
• Cell biology (slide 8): UCE (Universal Cell Embeddings) creates universal representations of cells. Input: the RNA expression of a single cell or nucleus. Output: a cell embedding. The figure shows genes sampled by expression, represented by protein language-model tokens, and fed to a transformer.
• Medicine (slide 9): "Foundation models for generalist medical artificial intelligence" (GMAI). One model is trained self-supervised on many modalities (text, audio, images, EHRs, signals, omics, graphs), uses medical knowledge sources (literature, clinical notes, knowledge graphs), and supports flexible interactions and applications (chatbots, note-taking, radiology reports, bedside decision support).

Slide 10 gives a definition (the answer of a chatbot to "define a foundation model"): a large-scale model trained on broad data that serves as a general-purpose model across many tasks (GPT-3, BERT, CLIP). It is characterized by:

1. Pretraining on large datasets with self-supervised or unsupervised learning.
2. Scalability: large architectures with billions or trillions of parameters.
3. Adaptability: after pretraining, fine-tuned or adapted to many downstream tasks with little task-specific data.
4. Transfer learning: knowledge from pretraining transfers to new domains.

Notice that graphs and knowledge graphs already appear inside the medical foundation model of slide 9 as one modality. This lecture asks whether graphs can have their own foundation model.

Slide 11 lists four characteristics of foundation models:

• They are pretrained architectures.
• Modern AI trains very large models on a huge amount of data.
• They can be used as zero-shot or few-shot learners.
• They generalize to new entities that were not observed during training.

The last point is the key one for this lecture. A shallow KG embedding (Lecture 10) has one learned vector per entity, so it cannot handle an entity it never saw.

Slide 12, graph foundation models: the challenges are multiple types of graphs (social networks, knowledge graphs, molecules in the pictures) and multiple types of tasks.

Slide 13 recalls PRODIGY (Lecture 14): a unified framework to learn graph tasks at multiple levels (node, edge, graph), agnostic of the graph type. Pipeline: pretraining data → data generation (Prompt Graphs) → model → optimize loss.

Slide 14 sets today's goal, towards foundation models for KGs:

• Build a model that learns embeddings for entities and relations of any KG.
• Train it in an unsupervised fashion.
• It should transfer knowledge between different KGs.
• Scalability of the training data and of the model is not discussed.

Slide 16 recalls the representation from Lecture 10. Edges in a KG are triples \((h, r, t)\): head \(h\) has relation \(r\) with tail \(t\). Key idea:

• Model entities and relations in an embedding space \(\mathbb{R}^k\), with shallow embeddings (note: no GNN is learned here).
• Given a true triple \((h, r, t)\), the embedding of \((h, r)\) should be close to the embedding of \(t\).
• Design questions: how to embed \((h, r)\)? How to define the score \(f_r(h, t)\)? The score should be high if \((h, r, t)\) exists and low otherwise.

Examples of KGs:

• Slide 17, bio KG. Node types: drug, disease, adverse event, protein, pathways. Relation types: has_func, causes, assoc, treats, is_a.
• Slide 18, bibliographic network. Node types: paper, title, author, conference, year. Relation types: pubWhere, pubYear, hasTitle, hasAuthor, cite (Paper is in the centre of the schema).
• Slide 19, e-commerce graph. Node types: User, Item, Query, Location, … Edge types: Purchase, Visit, Guide, Search, … Different node types can have different feature spaces. The figure links three users to items (shoes, a handbag) and queries ("Shoes", "Ladies Bag", "Hand Bag") with Search, Click and Guide edges.

Slides 20–22 compare a KG with natural language, using the sentence "User searches for shoes, User buys hand-bag":

• In language, all items are words or tokens of the same type: a homogeneous vocabulary. In the KG, "User", "shoes", "hand-bag" are entities and "searches", "buys" are relations: two modalities, a heterogeneous vocabulary.
• In language, tokens are connected sequentially. In a KG, entities have an underlying complex graph structure.

Why this matters: an LLM's vocabulary is fixed and shared across all texts, so transfer is natural. A new KG brings a new vocabulary of entities and relations, with no fixed order. A KG foundation model must handle a vocabulary it has never seen.

Slide 23 gives the recipe for a KG foundation model, which the rest of the lecture fills in:

• Learn embeddings for entities and relations of any KG → double equivariant architectures.
• Process the graph structure between entities → use graph neural networks.
• Find a canonical task for unsupervised training → knowledge graph completion.

Slide 25 recalls the KG completion task: given an enormous KG, for a given (head, relation), predict the missing tails. The example graph has authors (J.R.R. Tolkien, Stephen King, J.K. Rowling, Alan Poe, C.S. Lewis, Lloyd Alexander), genres (Science Fiction, Fantasy, Tragicomedy) and relations genre and Influence; the task is to predict "Science Fiction" for ("J.K. Rowling", "genre").

Slide 26: the simplest encoder is an embedding lookup. The matrix \(\mathbf{Z}\) has one column per node; its height is the embedding dimension.

Slide 27 recalls the KG embedding models and which relation patterns each can express:

Slide 28 summarizes KG embedding methods:

• Structure-agnostic shallow encoders learn embeddings for entities and relations; these embeddings are the trainable weights.
• The structure of the KG is used only in the loss function (which triples are positive).
• They can be used on a single KG only; they cannot transfer knowledge between KGs.

Slides 29–30 make the limits concrete. Slide 29: in a bio KG, what if a new drug comes in? It has no learned vector, so we need to retrain the shallow encoder. Slide 30: what if we want to transfer knowledge from one KG to another, when the KGs have the same relations? The two e-commerce graphs share Search/Click/Guide, but the second one has new items (books, an Xbox, the game NBA2K). Shallow embeddings of the first graph say nothing about these new items.

This is transductive link prediction: train and test on the same set of entities.

Slide 32 solves the case of slide 30 (transfer between KGs with the same relations). Observe that a KG is a heterogeneous graph, and use a heterogeneous GNN, for example R-GCN or the Heterogeneous Graph Transformer. A GNN computes an entity's embedding from its neighbourhood, so a new entity gets an embedding as soon as it has edges.

Slides 33–35 recall R-GCN (Lecture 9) on the example graph with nodes A–F and relations \(r_1, r_2, r_3\) (A is the target node). What if the graph has multiple relation types? Use different neural network weights for different relation types: \(\mathbf{W}_{r_1}\), \(\mathbf{W}_{r_2}\), \(\mathbf{W}_{r_3}\) (coloured boxes on slide 34). Slide 35 draws A's computation graph: each neighbour's message passes through the box of its relation, then all messages are aggregated into A.

Slide 36 recalls the GCN layer (Kipf and Welling, ICLR 2017): \[ \mathbf{h}_v^{(l)} = \sigma\Big( \sum_{u\in N(v)} \mathbf{W}^{(l)} \frac{\mathbf{h}_u^{(l-1)}}{|N(v)|} \Big), \] and with a self-loop: \[ \mathbf{h}_v^{(l)} = \sigma\Big( \sum_{u\in N(v)} \mathbf{W}^{(l)} \frac{\mathbf{h}_u^{(l-1)}}{|N(v)|} + \mathbf{W}^{(l)} \mathbf{h}_v^{(l-1)} \Big). \]

Slide 37: introduce one set of weights for each relation type (rel_1 … rel_N) plus a weight for the self-loop. Slide 38 gives the R-GCN definition: \[ \mathbf{h}_v^{(l+1)} = \sigma\Big( \sum_{r\in R} \sum_{u\in N_v^r} \frac{1}{c_{v,r}} \mathbf{W}_r^{(l)} \mathbf{h}_u^{(l)} + \mathbf{W}_0^{(l)} \mathbf{h}_v^{(l)} \Big). \] As message + aggregation:

• Message from each neighbour of a given relation, normalized by the node's degree for that relation, \(c_{v,r} = |N_v^r|\): \(\mathbf{m}_{u,r}^{(l)} = \frac{1}{c_{v,r}} \mathbf{W}_r^{(l)} \mathbf{h}_u^{(l)}\).
• Self-loop message: \(\mathbf{m}_v^{(l)} = \mathbf{W}_0^{(l)} \mathbf{h}_v^{(l)}\).
• Aggregation: sum the messages from neighbours and the self-loop, then apply the activation: \(\mathbf{h}_v^{(l+1)} = \sigma\big(\mathrm{Sum}(\{\mathbf{m}_{u,r}^{(l)}, u\in N(v)\} \cup \{\mathbf{m}_v^{(l)}\})\big)\).

Slide 39 recalls how R-GCN does link prediction. Assume \((E, r_3, A)\) is a training supervision edge and all other edges are training message edges. Use R-GCN to score \((E, r_3, A)\):

• Take the final-layer embeddings of E and A: \(\mathbf{h}_E^{(L)}, \mathbf{h}_A^{(L)} \in \mathbb{R}^d\).
• Use a relation-specific score function \(f_r: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}\). One example: \(f_{r_3}(\mathbf{h}_E, \mathbf{h}_A) = \mathbf{h}_E^\top \mathbf{W}_{r_3} \mathbf{h}_A\), with \(\mathbf{W}_{r_3} \in \mathbb{R}^{d\times d}\).

Slide 40 summarizes heterogeneous GNNs:

• Structure-aware GNN encoders learn embeddings for entities.
• Structure-agnostic shallow encoders still learn embeddings for relations: the relation embeddings (\(\mathbf{W}_r\)) are trainable weights.
• The structure of the KG is also used in the loss function.
• R-GCN can do inductive link prediction across new entities.
• But it cannot transfer knowledge between KGs with different relation types.

So we have climbed one step. Entities moved from "trainable weights" to "computed from structure". Relations have not moved yet. Check yourself: in R-GCN, what happens at test time if an edge has a relation type \(r_{\text{new}}\) never seen in training? (There is no \(\mathbf{W}_{r_{\text{new}}}\), neither for message passing nor for scoring.)

Slide 42 poses the hardest case: transfer knowledge from one KG to another when the KGs have different entities and different relations.

Slides 43–44 show it on examples. Slide 43: we train a model on one KG (the authors-and-genres graph of slide 25: genre, Influence). Slide 44: given the trained model, we want to do completion on a completely different KG: a movie graph with Olivia Colman, Emma Stone, Ryan Gosling, Yorgos Lanthimos, films (The Favorite, Poor Things, LaLa Land), countries (UK, Greece, USA, Canada), professions (actor, director), and relations Born in, Acted in, directed, costarred, Profession. The task: predict the tail "actor" for ("R. Gosling", "profession"). None of these entities or relations appeared in training.

Slide 45 recalls the easier case (same relations, slide 30), solved by heterogeneous GNNs. Slide 47 gives the solution for the hard case: build double equivariant models, for example InGram (Lee, Chung, Whang, "InGram: Inductive Knowledge Graph Embedding via Relation Graphs", ICML 2023).

What does "double equivariant" mean? If you rename (permute) the entities, the outputs are renamed the same way; and if you rename (permute) the relation types, the outputs are also renamed the same way. A model with a weight per named relation is not equivariant to renaming relations, because the name picks the weight. A model that treats relations only through their pattern of use is.

Slide 48 gives the idea by analogy:

• We achieved inductive link prediction on new entities by going from structure-agnostic shallow encoders to structure-aware GNN encoders that learn embeddings for entities.
• The structure of the KG moved from the loss function only to the loss function and the model.
• Let's do the same for relations: use the structure of relations to generate relation embeddings.

Intuition: in the movie KG, the model has never seen the word "costarred". But it can see that "costarred" connects people who both "Acted in" the same films. How a relation sits among other relations is information that transfers.

Slide 49 shows the InGram pipeline (figure from the authors' slides). Training time, left to right:

1. Training graph: a KG with relations \(r_1, \dots, r_5\).
2. Build the relation graph of the training graph: one node per relation (r1–r5), edges between related relations.
3. Relation-level aggregation: a GNN over the relation graph computes relation embeddings.
4. Entity-level aggregation: a GNN over the original KG computes entity embeddings, using the relation embeddings.
5. Loss optimization: score (head, relation, tail).

Inference time: a new graph with different relations (\(r_6, \dots, r_9\)). Build its relation graph, then apply the same learned weights (blue arrows) for relation-level aggregation, entity-level aggregation and scoring, and predict missing links (dashed orange edges r7, r8, r9). The new component is the relation graph.

Slide 50 defines three kinds of matrices (the slide says "tree", meaning three):

• \(E_h\): head-entities × relations adjacency, \(E_h[i,j] = 1\) if entity \(i\) is a head in relation \(j\).
• \(E_t\): tail-entities × relations adjacency, \(E_t[i,j] = 1\) if entity \(i\) is a tail in relation \(j\).
• \(D_h, D_t\): diagonal entity × entity matrices with the entity degrees as heads and as tails.

Slide 51: the adjacency matrix of relations is \[ A = E_h^\top D_h^{-2} E_h + E_t^\top D_t^{-2} E_t . \] \(A\) is a weighted adjacency matrix: each edge has a weight that tells how important it is. In practice we are looking for entities that serve as heads and/or tails for relation pairs.

Read the formula entry by entry: \(A[r_i, r_j] = \sum_{e} \frac{E_h[e,r_i]E_h[e,r_j]}{\deg_h(e)^2} + \sum_{e} \frac{E_t[e,r_i]E_t[e,r_j]}{\deg_t(e)^2}\). Two relations are close if many entities are heads of both or tails of both. Each shared entity contributes \(1/\deg^2\), so a hub entity that touches every relation adds almost nothing, while a specific entity adds a lot (the same idea as down-weighting common words in TF-IDF).

Slides 52–56 compute relation-graph weights \(\beta\) on the movie KG of slide 44. For each pair of relations, keep only the edges of those two relations and look for entities that serve as heads and/or tails for both.

• Slide 53, Born in and Acted in: the shared heads are Olivia Colman and Emma Stone. Their contributions give \[ \beta = \left(\tfrac{1}{3}\right)^2 + \left(\tfrac{1}{7}\right)^2 = 0.13 . \] (Olivia Colman has 3 outgoing edges in the full graph: Born in, Acted in, Profession; Emma Stone is a hub with 7.)
• Slide 54, Acted in and costarred: three entities contribute, \[ \beta = \left(\tfrac{1}{3}\right)^2 + \left(\tfrac{1}{7}\right)^2 + \left(\tfrac{1}{2}\right)^2 = 0.38 . \]
• Slide 55, directed and Acted in: the shared tails are the films The Favorite (3 incoming edges: two "Acted in", one "directed") and Poor Things (2 incoming edges), \[ \beta = \left(\tfrac{1}{3}\right)^2 + \left(\tfrac{1}{2}\right)^2 = 0.36 . \]
• Slide 56, costarred and directed: no entity plays the same role in both, so \(\beta = 0\).

Slide 57 shows the resulting relation graph: five nodes (directed, costarred, Acted in, Profession, Born in) with weighted edges. It encodes the proximity between relations. The weights shown: costarred–Acted in 0.38, directed–Acted in 0.36, Born in–costarred 0.27, Born in–Profession 0.17, Born in–Acted in 0.13, Profession–Acted in 0.13, Profession–costarred 0.13, and 0.06 for directed–Born in and directed–Profession.

Notice that the relation graph does not depend on the names of relations, only on how entities connect to them. That is exactly why a model built on it can handle a KG whose relations it never saw.

Slide 58 repeats the InGram pipeline: next is relation-level message passing on the relation graph. It uses graph attention, so slides 59–62 recall GAT (Lecture 4).

Slide 59: the GAT layer is \[ \mathbf{h}_v^{(l)} = \sigma\Big( \sum_{u\in N(v)} \alpha_{vu} \mathbf{W}^{(l)} \mathbf{h}_u^{(l-1)} \Big), \] with attention weights \(\alpha_{vu}\). Not all of a node's neighbours are equally important. Attention is inspired by cognitive attention: \(\alpha_{vu}\) focuses on the important parts of the input and fades out the rest. The network should devote more computing power to that small but important part, and which part matters depends on the context and is learned in training.

Slide 60: let \(\alpha_{vu}\) be a byproduct of an attention mechanism \(a\). First, \(a\) computes attention coefficients across pairs of nodes from their messages: \[ e_{vu} = a\big(\mathbf{W}^{(l)}\mathbf{h}_u^{(l-1)}, \mathbf{W}^{(l)}\mathbf{h}_v^{(l-1)}\big). \] \(e_{vu}\) indicates the importance of \(u\)'s message to \(v\).

Slide 61: the approach is agnostic to the choice of \(a\). For example, a single-layer network: concatenate the two messages and apply a linear layer, \(e_{AB} = \mathrm{Linear}(\mathrm{Concat}(\mathbf{W}^{(l)}\mathbf{h}_A^{(l-1)}, \mathbf{W}^{(l)}\mathbf{h}_B^{(l-1)}))\). The parameters of \(a\) are trained jointly with the weight matrices, end to end.

Slide 62: normalize with a softmax so that \(\sum_{u\in N(v)} \alpha_{vu} = 1\): \[ \alpha_{vu} = \frac{\exp(e_{vu})}{\sum_{k\in N(v)} \exp(e_{vk})}, \] then take the weighted sum. For node A with neighbours B, C, D: \(\mathbf{h}_A^{(l)} = \sigma(\alpha_{AB}\mathbf{W}^{(l)}\mathbf{h}_B^{(l-1)} + \alpha_{AC}\mathbf{W}^{(l)}\mathbf{h}_C^{(l-1)} + \alpha_{AD}\mathbf{W}^{(l)}\mathbf{h}_D^{(l-1)})\).

Slide 63, InGram's relation attention mechanism. The nodes are now relations, with embeddings \(\mathbf{z}\). The only change is a term \(\beta_{vu}\) inside the softmax: \[ \alpha_{vu} = \frac{\exp(e_{vu} + \beta_{vu})}{\sum_{k\in N(v)} \exp(e_{vk} + \beta_{vk})}, \qquad \mathbf{z}_v^{(l)} = \sigma\Big(\sum_{u\in N(v)} \alpha_{vu} \mathbf{W}^{(l)} \mathbf{z}_u^{(l-1)}\Big). \] \(\beta_{vu}\) comes from the relation-graph weight between relations \(v\) and \(u\) (slides 52–57). So a relation listens more to relations that share many specific entities with it.

In the InGram paper, the affinity values are grouped into bins and each bin gets a learnable bias; the slides simplify this to "add \(\beta\)". Either way, the structural affinity steers the attention.

Slide 64 repeats the pipeline: next is entity-level message passing on the original KG, now that every relation \(r\) has an embedding \(\mathbf{z}_r\).

Slide 65, the entity attention mechanism. Two terms are computed and added: \[ \mathbf{h}_v^{\dagger(l+1)} = \sigma\Big( \sum_{r\in R} \sum_{u\in N_v^r} \mathbf{W}_1^{(l)} \mathbf{h}_u^{(l)} + \mathbf{W}_0^{(l)} \mathbf{h}_v^{(l)} \Big), \] \[ \mathbf{h}_v^{*(l+1)} = \sigma\Big( \sum_{r\in R} \sum_{u\in N_v^r} \alpha_{urv} \big( \mathbf{W}_3^{(l)} \mathbf{z}_r^{(l)} + \mathbf{W}_2^{(l)} \mathbf{h}_u^{(l)} \big) \Big), \] \[ \mathbf{h}_v^{(l+1)} = \mathbf{h}_v^{\dagger(l+1)} + \mathbf{h}_v^{*(l+1)} . \]

• The first term looks like R-GCN, but with one shared \(\mathbf{W}_1\) for all relations instead of a \(\mathbf{W}_r\) per relation. That is what makes it work for unseen relations.
• The second term brings in the relation: each message from neighbour \(u\) via relation \(r\) contains the relation embedding \(\mathbf{z}_r\), and is weighted by an attention \(\alpha_{urv}\) that depends on the triple.

So the entity update depends on the relation update: the relation-level GNN runs first, and its output feeds every entity message. All weights (\(\mathbf{W}_0, \dots, \mathbf{W}_3\), attention) are shared across relations, so the model never needs a parameter tied to a relation's name.

Slide 66 repeats the pipeline: next is the link-prediction head. Slides 67–68 (the same content) define it:

• Final embeddings: \(\mathbf{z}_k = \mathbf{M}\mathbf{z}_k^{(L)}\) for relation \(k\), and \(\mathbf{h}_i = \hat{\mathbf{M}}\mathbf{h}_i^{(\hat{L})}\) for entity \(i\) (a linear map of the last layer of each GNN; \(L\) and \(\hat L\) are the numbers of relation and entity layers).
• Scoring function, a DistMult-style score whose relation vector is computed, not looked up: \[ f(v_i, r_k, v_j) = \mathbf{h}_i^\top \,\mathrm{diag}(\overline{\mathbf{W}}\mathbf{z}_k)\, \mathbf{h}_j . \] The small figure shows head and tail vectors combined with the relation vector into one score.
• Loss, a margin (hinge) loss over true triples \((v_i, r_k, v_j) \in \mathcal{T}_{\text{tr}}\) and corrupted triples \((\mathring{v}_i, r_k, \mathring{v}_j) \in \mathring{\mathcal{T}}_{\text{tr}}\): \[ \sum_{(v_i,r_k,v_j)\in\mathcal{T}_{\text{tr}}} \ \sum_{(\mathring{v}_i,r_k,\mathring{v}_j)\in\mathring{\mathcal{T}}_{\text{tr}}} \max\big(0,\ \gamma - f(v_i,r_k,v_j) + f(\mathring{v}_i,r_k,\mathring{v}_j)\big). \] A true triple should score at least \(\gamma\) higher than a corrupted one (head or tail replaced).

Slide 69 summarizes InGram, completing the ladder of slides 28 and 40:

• Structure-aware GNN encoders learn embeddings for entities.
• Structure-aware GNN encoders learn embeddings for relations.
• The structure of the KG is also used in the loss function.
• InGram can do inductive link prediction across new entities and new relations. It is a model we can use to train a foundation model.

Check yourself: fill the table. Shallow embeddings: entities = weights, relations = weights. R-GCN: entities = computed, relations = weights. InGram: entities = computed, relations = computed.

Slide 70, the experiments: create different datasets from NELL, Wikidata and Freebase, and test InGram in transductive and inductive link prediction. The metric is MRR (mean reciprocal rank of the true tail; higher is better).

Slide 71 (titled "Transductive Link Prediction"): bar charts of MRR on NL-0 and NELL-995-v1 for 14 methods, grouped by colour (GraIL, CoMPILE, SNRI, INDIGO, RMPI in orange; BLP, QBLP, RAILD in yellow; NeuralLP, DRUM, NBFNet, RED-GNN in green; CompGCN, NodePiece in grey) and InGram in blue. InGram has the highest MRR on both: about 0.27 on NL-0 (NBFNet is next at about 0.26) and about 0.73 on NELL-995-v1 (NodePiece is next at about 0.67).

Slide 72, "Inductive Link Prediction", on NL-100, WK-100 and FB-100, where the test KG has new entities and new relations:

• NL-100: InGram about 0.31; the best baseline, RMPI, about 0.22.
• WK-100: InGram about 0.11; RED-GNN about 0.10; most others near zero. Several subgraph-based methods (GraIL, CoMPILE, SNRI, INDIGO, RMPI) have no bar.
• FB-100: InGram about 0.22; RED-GNN about 0.12.

(Values are read from the bar charts.) The gap is largest exactly in the setting InGram was designed for.

Slide 73, summary:

• We introduced a model that learns embeddings for entities and relations of any KG.
• The model (InGram) is trained in an unsupervised fashion for KG completion.
• The model can transfer knowledge between different KGs.
• This is a step towards foundation models for KGs.

Remember slide 14: scalability was left out. Pretraining one InGram-style model on many KGs at once, and testing zero-shot on new ones, is the next step (this is what later work such as ULTRA does).