Graph Learning

Notebooks CS224W companions Study guideLive marimo ↗

L10 · Knowledge graph embeddings

This is a read-only view of the notebook source: text and code, no outputs. To run it, open the live marimo server when the workstation tunnel is up, or run uv run marimo edit notebooks/cs224w/L10_knowledge_graphs.py.

import warnings

warnings.filterwarnings("ignore", category=FutureWarning)
# harmless Intel-GPU probe message printed by torch on this machine
warnings.filterwarnings("ignore", message=".*Level Zero.*")

import hashlib
import json
import time

import marimo as mo
from gl.nb import reveal
import numpy as np
import pandas as pd
import networkx as nx
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch_geometric.nn.kge import ComplEx, DistMult, RotatE, TransE

SEED = 0
rng = np.random.default_rng(SEED)
torch.manual_seed(SEED)
# several open notebooks share the CPU; more threads = contention
torch.set_num_threads(4)
# shared datasets (Cora etc. are cached here) and this folder's disk cache
DATA_DIR = mo.notebook_dir().parents[1] / "data"
CACHE_DIR = mo.notebook_dir() / "cache"
CACHE_DIR.mkdir(exist_ok=True)
plt.rcParams.update(
    {
        "figure.dpi": 100,
        "axes.spines.top": False,
        "axes.spines.right": False,
    }
)

L10 · Knowledge graph embeddings

CS224W Machine Learning with Graphs · hands-on companion to Lecture 10

Audience: you did L09 (relation types, ranking metrics). Prerequisites: vectors, complex numbers (\(e^{i\theta}\)), the idea of a margin / logistic loss.

By the end you can:

In short. A KG stores facts as triples (Obama, nationality, USA) and is badly incomplete. KG completion: given \((h,r,?)\), rank the tails. Give every entity and relation an embedding and a score \(f_r(h,t)\). TransE: a relation is a translation \(\mathbf h+\mathbf r\approx\mathbf t\). TransR: project into a relation space first. DistMult: a diagonal bilinear form. ComplEx: complex numbers. RotatE: a rotation in the complex plane. Each geometry can or cannot express certain patterns; which model your KG needs depends on which patterns it contains.

Outline

  1. Score functions by hand — TransE's Obama example; symmetry with TransE vs. RotatE.
  2. Relation patterns — the expressiveness table.
  3. A family KG with known patterns — train TransE, DistMult, ComplEx, RotatE (PyG) and measure per-relation filtered MRR.
  4. Exercises, pitfall, research links.

1 · Score functions

model entity / relation score \(f_r(h,t)\) (higher = more plausible)
TransE \(\mathbf h,\mathbf r,\mathbf t\in\mathbb R^d\) \(-\lVert\mathbf h+\mathbf r-\mathbf t\rVert\)
TransR \(\mathbf h,\mathbf t\in\mathbb R^d\), \(\mathbf r\in\mathbb R^k\), \(M_r\in\mathbb R^{k\times d}\) \(-\lVert M_r\mathbf h+\mathbf r-M_r\mathbf t\rVert\)
DistMult \(\mathbb R^d\) \(\langle\mathbf h,\mathbf r,\mathbf t\rangle=\sum_i h_ir_it_i\)
ComplEx \(\mathbb C^d\) \(\text{Re}\,\langle\mathbf h,\mathbf r,\bar{\mathbf t}\rangle\)
RotatE \(\mathbb C^d\), \(\lvert r_i\rvert=1\) \(-\lVert\mathbf h\circ\mathbf r-\mathbf t\rVert\)

Worked example (TransE, 2-D). \(\mathbf h_{\text{Obama}}=(0,0)\), candidates USA \((2,1)\), France \((3,-1)\), Kenya \((1.5,1.5)\). With \(\mathbf r_{\text{nationality}}=(2,1)\): distances USA 0, Kenya 0.71, France 2.24.

def score_transe(h, r, t):
    return -np.linalg.norm(h + r - t, axis=-1)

def score_transr(h, r, t, M):
    return -np.linalg.norm(h @ M.T + r - t @ M.T, axis=-1)

def score_distmult(h, r, t):
    return (h * r * t).sum(-1)

def score_complex(h, r, t):
    return np.real((h * r * np.conj(t)).sum(-1))

def score_rotate(h, theta, t):
    return -np.abs(h * np.exp(1j * theta) - t).sum(-1)

obama = np.array([0.0, 0.0])
countries = {
    "USA": np.array([2.0, 1.0]),
    "France": np.array([3.0, -1.0]),
    "Kenya": np.array([1.5, 1.5]),
}
rx = mo.ui.slider(-3, 3, step=0.25, value=2, label="r_x", show_value=True)
ry = mo.ui.slider(-3, 3, step=0.25, value=1, label="r_y", show_value=True)
mo.vstack(
    [
        mo.md(
            "**Try it:** move the relation vector $\\mathbf r$; the predicted tail is the entity closest to $\\mathbf h+\\mathbf r$."
        ),
        mo.hstack([rx, ry], justify="start"),
    ]
)
_r = np.array([rx.value, ry.value])
_scores = {c: score_transe(obama, _r, v) for c, v in countries.items()}
_ranked = sorted(_scores, key=_scores.get, reverse=True)
_fig, _ax = plt.subplots(figsize=(4.2, 3.2))
_ax.scatter(*obama, s=80, c="k")
_ax.annotate(
    "Obama (h)",
    obama,
    xytext=(4, -10),
    textcoords="offset points",
    fontsize=8,
)
for _c, _v in countries.items():
    _ax.scatter(*_v, s=70)
    _ax.annotate(
        _c, _v, xytext=(4, 4), textcoords="offset points", fontsize=8
    )
_ax.annotate(
    "",
    xy=obama + _r,
    xytext=obama,
    arrowprops=dict(arrowstyle="->", color="crimson", lw=2),
)
_ax.scatter(*(obama + _r), marker="x", c="crimson", s=80)
_ax.set_xlim(-3.5, 4)
_ax.set_ylim(-3.5, 3.5)
_ax.set_aspect("equal")
_ax.set_title("TransE: h + r ≈ t", fontsize=9)
mo.hstack(
    [
        _ax,
        mo.md(
            "| rank | tail | score −‖h+r−t‖ |\n|---|---|---|\n"
            + "\n".join(
                f"| {i + 1} | {c} | {_scores[c]:.2f} |"
                for i, c in enumerate(_ranked)
            )
        ),
    ],
    widths=[3, 2],
)

Symmetry test. "roommate" must hold both ways. TransE: if \(\mathbf h+\mathbf r=\mathbf t\) then \(\mathbf t+\mathbf r=\mathbf h+2\mathbf r\neq\mathbf h\) unless \(\mathbf r=0\), and \(\mathbf r=0\) would make every entity its own roommate. RotatE: a half-turn \(r=e^{i\pi}=-1\) is its own inverse.

_h, _t = (
    np.array([1.0 + 0.5j]),
    np.array([-1.0 - 0.5j]),
)  # t = -h: a "half-turn" pair
_h_re, _r_re = np.array([0.0, 0.0]), np.array([1.0, 0.0])
_t_re = _h_re + _r_re
pd.DataFrame(
    [
        {
            "model": "TransE, r = (1, 0)",
            "score(h, r, t)": score_transe(_h_re, _r_re, _t_re),
            "score(t, r, h)": score_transe(_t_re, _r_re, _h_re),
        },
        {
            "model": "RotatE, θ = π",
            "score(h, r, t)": score_rotate(_h, np.pi, _t),
            "score(t, r, h)": score_rotate(_t, np.pi, _h),
        },
        {
            "model": "DistMult (always symmetric)",
            "score(h, r, t)": score_distmult(
                np.array([1.0, 2.0]),
                np.array([0.5, 1.0]),
                np.array([2.0, 1.0]),
            ),
            "score(t, r, h)": score_distmult(
                np.array([2.0, 1.0]),
                np.array([0.5, 1.0]),
                np.array([1.0, 2.0]),
            ),
        },
    ]
).round(3)

TransE scores \((h,r,t)\) perfectly (0) but the reverse \((t,r,h)\) at −2: it cannot be symmetric. RotatE with \(\theta=\pi\) scores both directions perfectly. DistMult is symmetric for every relation — \(\sum_i h_ir_it_i\) doesn't care about order — which is a feature for "spouse" and a bug for "parent of".

2 · Relation patterns: the expressiveness table (lecture)

pattern example TransE TransR DistMult ComplEx RotatE
symmetric \(r(h,t)\Rightarrow r(t,h)\) spouse, sibling ✗ ✓ ✓ ✓ ✓
antisymmetric \(r(h,t)\Rightarrow\neg r(t,h)\) parent of ✓ ✓ ✗ ✓ ✓
inverse \(r_1(h,t)\Leftrightarrow r_2(t,h)\) parent of / child of ✓ ✓ ✗ ✓ ✓
composition \(r_1\circ r_2=r_3\) parent∘parent = grandparent ✓ ✓ ✗ ✗ ✓
1-to-N parent of several children ✗ ✓ ✓ ✓ ✗

"Can model" = representable. Whether training finds it is an empirical question. Let's test it.

3 · A family KG with known patterns

60 synthetic three-generation families (generated here): couples, children, grandchildren. Five relations: spouse (symmetric), parent_of (antisymmetric, 1-to-N), child_of (inverse of parent_of), grandparent_of (composition of parent_of with itself), sibling_of (symmetric, N-to-N). Hold out 10% of triples; train each PyG model (32-d, 300 epochs); report filtered MRR per relation (other true tails are removed from the candidate list).

KG_RELATIONS = [ · 67 lines
KG_RELATIONS = [
    "spouse",
    "parent_of",
    "child_of",
    "grandparent_of",
    "sibling_of",
]

def family_kg(n_families=60, seed=SEED):
    r = np.random.default_rng(seed)
    triples, pid = set(), 0

    def add(h, rel, t):
        triples.add((h, KG_RELATIONS.index(rel), t))

    for _ in range(n_families):
        ga, gb = pid, pid + 1
        pid += 2  # grandparents
        add(ga, "spouse", gb)
        add(gb, "spouse", ga)
        kids = []
        for _ in range(int(r.integers(2, 4))):
            c, partner = pid, pid + 1
            pid += 2
            kids.append(c)
            add(c, "spouse", partner)
            add(partner, "spouse", c)
            for p in (ga, gb):
                add(p, "parent_of", c)
                add(c, "child_of", p)
            grandkids = []
            for _ in range(int(r.integers(1, 4))):
                k = pid
                pid += 1
                grandkids.append(k)
                for p in (c, partner):
                    add(p, "parent_of", k)
                    add(k, "child_of", p)
                for gp in (ga, gb):
                    add(gp, "grandparent_of", k)
            for a in grandkids:
                for b in grandkids:
                    if a != b:
                        add(a, "sibling_of", b)
        for a in kids:
            for b in kids:
                if a != b:
                    add(a, "sibling_of", b)
    return np.array(sorted(triples)), pid

kg_triples, kg_n_entities = family_kg()
_perm = np.random.default_rng(SEED).permutation(len(kg_triples))
kg_test, kg_train = (
    kg_triples[_perm[: len(_perm) // 10]],
    kg_triples[_perm[len(_perm) // 10 :]],
)
kg_known = {}
for _h, _r, _t in kg_triples:
    kg_known.setdefault((_h, _r), set()).add(_t)
mo.md(
    f"**{len(kg_triples):,} triples**, {kg_n_entities} people; per relation: "
    + ", ".join(
        f"{n} {c}"
        for n, c in zip(KG_RELATIONS, np.bincount(kg_triples[:, 1]))
    )
    + f". Train {len(kg_train):,} / test {len(kg_test):,}."
)
def train_kge(cls, epochs=300, dim=32, seed=SEED, **kw): · 57 lines
def train_kge(cls, epochs=300, dim=32, seed=SEED, **kw):
    torch.manual_seed(seed)
    model = cls(
        num_nodes=kg_n_entities,
        num_relations=len(KG_RELATIONS),
        hidden_channels=dim,
        **kw,
    )
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    h, r, t = (torch.from_numpy(kg_train[:, i]) for i in range(3))
    loader = model.loader(
        head_index=h,
        rel_type=r,
        tail_index=t,
        batch_size=512,
        shuffle=True,
    )
    for _ in range(epochs):
        for (
            hb,
            rb,
            tb,
        ) in (
            loader
        ):  # PyG samples corrupted (negative) triples inside model.loss
            opt.zero_grad()
            model.loss(hb, rb, tb).backward()
            opt.step()
    return model

def filtered_mrr_per_relation(model):
    out = {}
    all_t = torch.arange(kg_n_entities)
    with torch.no_grad():
        for ri, name in enumerate(KG_RELATIONS):
            rr = []
            for h, _, t in kg_test[kg_test[:, 1] == ri]:
                s = model(
                    torch.full((kg_n_entities,), int(h)),
                    torch.full((kg_n_entities,), ri),
                    all_t,
                )
                s[[x for x in kg_known[(h, ri)] if x != t]] = -float(
                    "inf"
                )  # filtered setting
                rr.append(1.0 / (1 + (s > s[t]).sum().item()))
            out[name] = np.mean(rr)
    return out

kge_models = {
    "TransE": (TransE, {"margin": 1.0}),
    "DistMult": (DistMult, {}),
    "ComplEx": (ComplEx, {}),
    "RotatE": (RotatE, {"margin": 9.0}),
}
kge_run = mo.ui.run_button(label="Train the four KGE models (≈ 15 s)")
kge_run
mo.stop(
    not kge_run.value,
    mo.md(
        "Press the button to train TransE, DistMult, ComplEx and RotatE on the family KG."
    ),
)
kge_results = pd.DataFrame(
    {
        name: filtered_mrr_per_relation(train_kge(cls, **kw))
        for name, (cls, kw) in kge_models.items()
    }
).T
_patterns = [
    "symmetric",
    "antisym., 1-to-N",
    "inverse",
    "composition",
    "symmetric, N-to-N",
]
_fig, _ax = plt.subplots(figsize=(8.5, 2.8))
_im = _ax.imshow(
    kge_results.values, cmap="RdYlGn", vmin=0, vmax=1, aspect="auto"
)
_ax.set_xticks(
    range(len(KG_RELATIONS)),
    [f"{r}\n{p}" for r, p in zip(KG_RELATIONS, _patterns)],
    fontsize=8,
)
_ax.set_yticks(range(len(kge_results)), kge_results.index, fontsize=8)
for (_i, _j), _v in np.ndenumerate(kge_results.values):
    _ax.text(_j, _i, f"{_v:.2f}", ha="center", va="center", fontsize=9)
_ax.set_title(
    "filtered MRR on held-out triples (1.0 = true tail always ranked first)",
    fontsize=9,
)
plt.colorbar(_im, ax=_ax, fraction=0.03)
plt.tight_layout()
_fig
_rev = reveal( · 33 lines
_rev = reveal(
    r"""
Measured (one seed; numbers move by a few points between runs):

* **TransE** fails both **symmetric** relations (spouse 0.46, sibling 0.44) yet is near-perfect on parent/child (inverse: 0.96/0.99) and
  **grandparent (composition: 1.00)**, exactly the table's TransE column.
* **DistMult** is mediocre everywhere (0.59–0.71). Because it scores $(h,r,t)$ and $(t,r,h)$ identically, it cannot tell
  "A parent_of B" from "B parent_of A", and that confusion spills over to every relation that shares those entities.
* **ComplEx** is solid everywhere (0.85–0.91), including composition, which it "can't" represent in general; on this small KG it
  approximates it.
* **RotatE** is best on composition (1.00) and directional relations (0.94–0.95), and decent but not perfect on symmetric ones
  (0.74–0.76): representable (θ = π) but harder to reach by training.

""",
    predict="Using the cheat-sheet table, mark for TransE, DistMult, ComplEx and RotatE which relations (symmetric spouse/sibling, inverse parent/child, composed grandparent) each should fail. Then check.",
)
_post = mo.md(r"""
Reading the table is the skill: check which patterns your KG has *before* choosing a model.

---

## 4 · Exercises

**Exercise 1 (TransE by hand).** $\mathbf h=(1,1)$, $\mathbf r=(0,2)$; candidates A $(1,3)$, B $(2,2)$, C $(0,3)$. Which candidate is ranked
**second**? → `ex1_second` (a string).

**Exercise 2 (RotatE composition).** RotatE relation $r_k$ is a vector of angles $\boldsymbol\theta_k$. Write `compose(theta1, theta2)`
returning the angles of $r_3$ with $r_3(x)=r_2(r_1(x))$, wrapped to $[0, 2\pi)$.

**Exercise 3 (filtered rank).** The true tail scores 0.6; the other candidates score 0.9 (*also a known true tail*), 0.7 and 0.2.
Raw rank is 3. What is the **filtered** rank? → `ex3_rank`.
""")
mo.vstack([_rev, _post])
ex1_second = None · 5 lines
ex1_second = None
ex3_rank = None

def compose(theta1, theta2):
    return None
def _verdict(ok, answered): · 49 lines
def _verdict(ok, answered):
    if not answered:
        return mo.callout(mo.md("Not answered yet."), kind="neutral")
    return mo.callout(
        mo.md("Correct!" if ok else "Not yet — try again or open a hint."),
        kind="success" if ok else "warn",
    )

def _check_compose():
    try:
        g = np.random.default_rng(3)
        t1, t2 = g.uniform(0, 2 * np.pi, 5), g.uniform(0, 2 * np.pi, 5)
        got = compose(t1, t2)
        if got is None:
            return _verdict(False, False)
        h = g.normal(size=5) + 1j * g.normal(size=5)
        ok = np.allclose(
            h * np.exp(1j * t1) * np.exp(1j * t2),
            h * np.exp(1j * np.asarray(got)),
        ) and np.all(
            (np.asarray(got) >= 0) & (np.asarray(got) < 2 * np.pi)
        )
        return _verdict(ok, True)
    except Exception as e:
        return mo.callout(mo.md(f"Error: `{e}`"), kind="danger")

mo.vstack(
    [
        mo.md("**Exercise 1**"),
        _verdict(ex1_second == "C", ex1_second is not None),
        mo.md("**Exercise 2**"),
        _check_compose(),
        mo.md("**Exercise 3**"),
        _verdict(ex3_rank == 2, ex3_rank is not None),
        mo.accordion(
            {
                "Solution 1": mo.md(
                    r"$\mathbf h+\mathbf r=(1,3)$. Distances: A 0, C 1, B $\sqrt2\approx1.41$ → order A, **C**, B."
                ),
                "Solution 2": mo.md(
                    "```python\ndef compose(theta1, theta2):\n    return np.mod(np.asarray(theta1) + np.asarray(theta2), 2 * np.pi)\n```\nRotations compose by adding angles; this is why RotatE models composition (grandparent = parent ∘ parent: θ_gp = 2 θ_p)."
                ),
                "Solution 3": mo.md(
                    "Remove the other known true tail (0.9): only 0.7 beats 0.6 → filtered rank **2** (raw rank 3)."
                ),
            }
        ),
    ]
)

5 · Pitfall: raw (unfiltered) ranking punishes correct answers

A person has several children. When the test triple is (Ana, parent_of, Bo), the model ranking Ana's other child Cy first is right, yet raw ranking counts it as an error. Same trained ComplEx, two protocols:

def mrr(model, rel, filtered): · 27 lines
def mrr(model, rel, filtered):
    rr, all_t = [], torch.arange(kg_n_entities)
    with torch.no_grad():
        for h, _, t in kg_test[kg_test[:, 1] == rel]:
            s = model(
                torch.full((kg_n_entities,), int(h)),
                torch.full((kg_n_entities,), rel),
                all_t,
            )
            if filtered:
                s[[x for x in kg_known[(h, rel)] if x != t]] = -float(
                    "inf"
                )
            rr.append(1.0 / (1 + (s > s[t]).sum().item()))
    return np.mean(rr)

_m = train_kge(ComplEx)
pd.DataFrame(
    [
        {
            "relation": KG_RELATIONS[r],
            "raw MRR": mrr(_m, r, False),
            "filtered MRR": mrr(_m, r, True),
        }
        for r in range(5)
    ]
).round(3)

Raw MRR understates every 1-to-N and N-to-N relation (several true tails compete for rank 1), while spouse (1-to-1) is unaffected. Fix: always report filtered metrics for KG completion, and say so. OGB's link-prediction benchmarks avoid the issue by fixing a set of sampled negatives per test edge.

6 · Connecting to your research (SNA on X/Twitter)

Summary

Next — L11 · GNNs for recommender systems: BPR, LightGCN, PinSage.