This is a read-only view of the notebook source: text and code, no outputs. To run it, open the live marimo server when the workstation tunnel is up, or run uv run marimo edit notebooks/cs224w/L10_knowledge_graphs.py.
import warnings warnings.filterwarnings("ignore", category=FutureWarning) # harmless Intel-GPU probe message printed by torch on this machine warnings.filterwarnings("ignore", message=".*Level Zero.*") import hashlib import json import time import marimo as mo from gl.nb import reveal import numpy as np import pandas as pd import networkx as nx import matplotlib.pyplot as plt import torch import torch.nn as nn import torch.nn.functional as F from torch_geometric.nn.kge import ComplEx, DistMult, RotatE, TransE SEED = 0 rng = np.random.default_rng(SEED) torch.manual_seed(SEED) # several open notebooks share the CPU; more threads = contention torch.set_num_threads(4) # shared datasets (Cora etc. are cached here) and this folder's disk cache DATA_DIR = mo.notebook_dir().parents[1] / "data" CACHE_DIR = mo.notebook_dir() / "cache" CACHE_DIR.mkdir(exist_ok=True) plt.rcParams.update( { "figure.dpi": 100, "axes.spines.top": False, "axes.spines.right": False, } )
L10 · Knowledge graph embeddings
CS224W Machine Learning with Graphs · hands-on companion to Lecture 10
Audience: you did L09 (relation types, ranking metrics). Prerequisites: vectors, complex numbers (\(e^{i\theta}\)), the idea of a margin / logistic loss.
By the end you can:
- score triples \((h,r,t)\) with TransE, TransR, DistMult, ComplEx, RotatE, by hand (the lecture's Obama example) and in code;
- reason about which relation patterns (symmetry, antisymmetry, inversion, composition, 1-to-N) each geometry can express;
- train PyG's KGE models on a synthetic family KG with known patterns and confirm the expressiveness table empirically, with filtered MRR / Hits@k.
In short. A KG stores facts as triples (Obama, nationality, USA) and is badly incomplete. KG completion: given \((h,r,?)\), rank the tails. Give every entity and relation an embedding and a score \(f_r(h,t)\). TransE: a relation is a translation \(\mathbf h+\mathbf r\approx\mathbf t\). TransR: project into a relation space first. DistMult: a diagonal bilinear form. ComplEx: complex numbers. RotatE: a rotation in the complex plane. Each geometry can or cannot express certain patterns; which model your KG needs depends on which patterns it contains.
Outline
- Score functions by hand — TransE's Obama example; symmetry with TransE vs. RotatE.
- Relation patterns — the expressiveness table.
- A family KG with known patterns — train TransE, DistMult, ComplEx, RotatE (PyG) and measure per-relation filtered MRR.
- Exercises, pitfall, research links.
1 · Score functions
| model | entity / relation | score \(f_r(h,t)\) (higher = more plausible) |
|---|---|---|
| TransE | \(\mathbf h,\mathbf r,\mathbf t\in\mathbb R^d\) | \(-\lVert\mathbf h+\mathbf r-\mathbf t\rVert\) |
| TransR | \(\mathbf h,\mathbf t\in\mathbb R^d\), \(\mathbf r\in\mathbb R^k\), \(M_r\in\mathbb R^{k\times d}\) | \(-\lVert M_r\mathbf h+\mathbf r-M_r\mathbf t\rVert\) |
| DistMult | \(\mathbb R^d\) | \(\langle\mathbf h,\mathbf r,\mathbf t\rangle=\sum_i h_ir_it_i\) |
| ComplEx | \(\mathbb C^d\) | \(\text{Re}\,\langle\mathbf h,\mathbf r,\bar{\mathbf t}\rangle\) |
| RotatE | \(\mathbb C^d\), \(\lvert r_i\rvert=1\) | \(-\lVert\mathbf h\circ\mathbf r-\mathbf t\rVert\) |
Worked example (TransE, 2-D). \(\mathbf h_{\text{Obama}}=(0,0)\), candidates USA \((2,1)\), France \((3,-1)\), Kenya \((1.5,1.5)\). With \(\mathbf r_{\text{nationality}}=(2,1)\): distances USA 0, Kenya 0.71, France 2.24.
def score_transe(h, r, t): return -np.linalg.norm(h + r - t, axis=-1) def score_transr(h, r, t, M): return -np.linalg.norm(h @ M.T + r - t @ M.T, axis=-1) def score_distmult(h, r, t): return (h * r * t).sum(-1) def score_complex(h, r, t): return np.real((h * r * np.conj(t)).sum(-1)) def score_rotate(h, theta, t): return -np.abs(h * np.exp(1j * theta) - t).sum(-1) obama = np.array([0.0, 0.0]) countries = { "USA": np.array([2.0, 1.0]), "France": np.array([3.0, -1.0]), "Kenya": np.array([1.5, 1.5]), } rx = mo.ui.slider(-3, 3, step=0.25, value=2, label="r_x", show_value=True) ry = mo.ui.slider(-3, 3, step=0.25, value=1, label="r_y", show_value=True) mo.vstack( [ mo.md( "**Try it:** move the relation vector $\\mathbf r$; the predicted tail is the entity closest to $\\mathbf h+\\mathbf r$." ), mo.hstack([rx, ry], justify="start"), ] )
_r = np.array([rx.value, ry.value]) _scores = {c: score_transe(obama, _r, v) for c, v in countries.items()} _ranked = sorted(_scores, key=_scores.get, reverse=True) _fig, _ax = plt.subplots(figsize=(4.2, 3.2)) _ax.scatter(*obama, s=80, c="k") _ax.annotate( "Obama (h)", obama, xytext=(4, -10), textcoords="offset points", fontsize=8, ) for _c, _v in countries.items(): _ax.scatter(*_v, s=70) _ax.annotate( _c, _v, xytext=(4, 4), textcoords="offset points", fontsize=8 ) _ax.annotate( "", xy=obama + _r, xytext=obama, arrowprops=dict(arrowstyle="->", color="crimson", lw=2), ) _ax.scatter(*(obama + _r), marker="x", c="crimson", s=80) _ax.set_xlim(-3.5, 4) _ax.set_ylim(-3.5, 3.5) _ax.set_aspect("equal") _ax.set_title("TransE: h + r ≈ t", fontsize=9) mo.hstack( [ _ax, mo.md( "| rank | tail | score −‖h+r−t‖ |\n|---|---|---|\n" + "\n".join( f"| {i + 1} | {c} | {_scores[c]:.2f} |" for i, c in enumerate(_ranked) ) ), ], widths=[3, 2], )
Symmetry test. "roommate" must hold both ways. TransE: if \(\mathbf h+\mathbf r=\mathbf t\) then \(\mathbf t+\mathbf r=\mathbf h+2\mathbf r\neq\mathbf h\) unless \(\mathbf r=0\), and \(\mathbf r=0\) would make every entity its own roommate. RotatE: a half-turn \(r=e^{i\pi}=-1\) is its own inverse.
_h, _t = ( np.array([1.0 + 0.5j]), np.array([-1.0 - 0.5j]), ) # t = -h: a "half-turn" pair _h_re, _r_re = np.array([0.0, 0.0]), np.array([1.0, 0.0]) _t_re = _h_re + _r_re pd.DataFrame( [ { "model": "TransE, r = (1, 0)", "score(h, r, t)": score_transe(_h_re, _r_re, _t_re), "score(t, r, h)": score_transe(_t_re, _r_re, _h_re), }, { "model": "RotatE, θ = π", "score(h, r, t)": score_rotate(_h, np.pi, _t), "score(t, r, h)": score_rotate(_t, np.pi, _h), }, { "model": "DistMult (always symmetric)", "score(h, r, t)": score_distmult( np.array([1.0, 2.0]), np.array([0.5, 1.0]), np.array([2.0, 1.0]), ), "score(t, r, h)": score_distmult( np.array([2.0, 1.0]), np.array([0.5, 1.0]), np.array([1.0, 2.0]), ), }, ] ).round(3)
TransE scores \((h,r,t)\) perfectly (0) but the reverse \((t,r,h)\) at −2: it cannot be symmetric. RotatE with \(\theta=\pi\) scores both directions perfectly. DistMult is symmetric for every relation — \(\sum_i h_ir_it_i\) doesn't care about order — which is a feature for "spouse" and a bug for "parent of".
2 · Relation patterns: the expressiveness table (lecture)
| pattern | example | TransE | TransR | DistMult | ComplEx | RotatE |
|---|---|---|---|---|---|---|
| symmetric \(r(h,t)\Rightarrow r(t,h)\) | spouse, sibling | ✗ | ✓ | ✓ | ✓ | ✓ |
| antisymmetric \(r(h,t)\Rightarrow\neg r(t,h)\) | parent of | ✓ | ✓ | ✗ | ✓ | ✓ |
| inverse \(r_1(h,t)\Leftrightarrow r_2(t,h)\) | parent of / child of | ✓ | ✓ | ✗ | ✓ | ✓ |
| composition \(r_1\circ r_2=r_3\) | parent∘parent = grandparent | ✓ | ✓ | ✗ | ✗ | ✓ |
| 1-to-N | parent of several children | ✗ | ✓ | ✓ | ✓ | ✗ |
"Can model" = representable. Whether training finds it is an empirical question. Let's test it.
3 · A family KG with known patterns
60 synthetic three-generation families (generated here): couples, children, grandchildren. Five relations: spouse (symmetric), parent_of (antisymmetric, 1-to-N), child_of (inverse of parent_of), grandparent_of (composition of parent_of with itself), sibling_of (symmetric, N-to-N). Hold out 10% of triples; train each PyG model (32-d, 300 epochs); report filtered MRR per relation (other true tails are removed from the candidate list).
KG_RELATIONS = [ · 67 lines
KG_RELATIONS = [ "spouse", "parent_of", "child_of", "grandparent_of", "sibling_of", ] def family_kg(n_families=60, seed=SEED): r = np.random.default_rng(seed) triples, pid = set(), 0 def add(h, rel, t): triples.add((h, KG_RELATIONS.index(rel), t)) for _ in range(n_families): ga, gb = pid, pid + 1 pid += 2 # grandparents add(ga, "spouse", gb) add(gb, "spouse", ga) kids = [] for _ in range(int(r.integers(2, 4))): c, partner = pid, pid + 1 pid += 2 kids.append(c) add(c, "spouse", partner) add(partner, "spouse", c) for p in (ga, gb): add(p, "parent_of", c) add(c, "child_of", p) grandkids = [] for _ in range(int(r.integers(1, 4))): k = pid pid += 1 grandkids.append(k) for p in (c, partner): add(p, "parent_of", k) add(k, "child_of", p) for gp in (ga, gb): add(gp, "grandparent_of", k) for a in grandkids: for b in grandkids: if a != b: add(a, "sibling_of", b) for a in kids: for b in kids: if a != b: add(a, "sibling_of", b) return np.array(sorted(triples)), pid kg_triples, kg_n_entities = family_kg() _perm = np.random.default_rng(SEED).permutation(len(kg_triples)) kg_test, kg_train = ( kg_triples[_perm[: len(_perm) // 10]], kg_triples[_perm[len(_perm) // 10 :]], ) kg_known = {} for _h, _r, _t in kg_triples: kg_known.setdefault((_h, _r), set()).add(_t) mo.md( f"**{len(kg_triples):,} triples**, {kg_n_entities} people; per relation: " + ", ".join( f"{n} {c}" for n, c in zip(KG_RELATIONS, np.bincount(kg_triples[:, 1])) ) + f". Train {len(kg_train):,} / test {len(kg_test):,}." )
def train_kge(cls, epochs=300, dim=32, seed=SEED, **kw): · 57 lines
def train_kge(cls, epochs=300, dim=32, seed=SEED, **kw): torch.manual_seed(seed) model = cls( num_nodes=kg_n_entities, num_relations=len(KG_RELATIONS), hidden_channels=dim, **kw, ) opt = torch.optim.Adam(model.parameters(), lr=0.01) h, r, t = (torch.from_numpy(kg_train[:, i]) for i in range(3)) loader = model.loader( head_index=h, rel_type=r, tail_index=t, batch_size=512, shuffle=True, ) for _ in range(epochs): for ( hb, rb, tb, ) in ( loader ): # PyG samples corrupted (negative) triples inside model.loss opt.zero_grad() model.loss(hb, rb, tb).backward() opt.step() return model def filtered_mrr_per_relation(model): out = {} all_t = torch.arange(kg_n_entities) with torch.no_grad(): for ri, name in enumerate(KG_RELATIONS): rr = [] for h, _, t in kg_test[kg_test[:, 1] == ri]: s = model( torch.full((kg_n_entities,), int(h)), torch.full((kg_n_entities,), ri), all_t, ) s[[x for x in kg_known[(h, ri)] if x != t]] = -float( "inf" ) # filtered setting rr.append(1.0 / (1 + (s > s[t]).sum().item())) out[name] = np.mean(rr) return out kge_models = { "TransE": (TransE, {"margin": 1.0}), "DistMult": (DistMult, {}), "ComplEx": (ComplEx, {}), "RotatE": (RotatE, {"margin": 9.0}), } kge_run = mo.ui.run_button(label="Train the four KGE models (≈ 15 s)") kge_run
mo.stop( not kge_run.value, mo.md( "Press the button to train TransE, DistMult, ComplEx and RotatE on the family KG." ), ) kge_results = pd.DataFrame( { name: filtered_mrr_per_relation(train_kge(cls, **kw)) for name, (cls, kw) in kge_models.items() } ).T _patterns = [ "symmetric", "antisym., 1-to-N", "inverse", "composition", "symmetric, N-to-N", ] _fig, _ax = plt.subplots(figsize=(8.5, 2.8)) _im = _ax.imshow( kge_results.values, cmap="RdYlGn", vmin=0, vmax=1, aspect="auto" ) _ax.set_xticks( range(len(KG_RELATIONS)), [f"{r}\n{p}" for r, p in zip(KG_RELATIONS, _patterns)], fontsize=8, ) _ax.set_yticks(range(len(kge_results)), kge_results.index, fontsize=8) for (_i, _j), _v in np.ndenumerate(kge_results.values): _ax.text(_j, _i, f"{_v:.2f}", ha="center", va="center", fontsize=9) _ax.set_title( "filtered MRR on held-out triples (1.0 = true tail always ranked first)", fontsize=9, ) plt.colorbar(_im, ax=_ax, fraction=0.03) plt.tight_layout() _fig
_rev = reveal( · 33 lines
_rev = reveal( r""" Measured (one seed; numbers move by a few points between runs): * **TransE** fails both **symmetric** relations (spouse 0.46, sibling 0.44) yet is near-perfect on parent/child (inverse: 0.96/0.99) and **grandparent (composition: 1.00)**, exactly the table's TransE column. * **DistMult** is mediocre everywhere (0.59–0.71). Because it scores $(h,r,t)$ and $(t,r,h)$ identically, it cannot tell "A parent_of B" from "B parent_of A", and that confusion spills over to every relation that shares those entities. * **ComplEx** is solid everywhere (0.85–0.91), including composition, which it "can't" represent in general; on this small KG it approximates it. * **RotatE** is best on composition (1.00) and directional relations (0.94–0.95), and decent but not perfect on symmetric ones (0.74–0.76): representable (θ = π) but harder to reach by training. """, predict="Using the cheat-sheet table, mark for TransE, DistMult, ComplEx and RotatE which relations (symmetric spouse/sibling, inverse parent/child, composed grandparent) each should fail. Then check.", ) _post = mo.md(r""" Reading the table is the skill: check which patterns your KG has *before* choosing a model. --- ## 4 · Exercises **Exercise 1 (TransE by hand).** $\mathbf h=(1,1)$, $\mathbf r=(0,2)$; candidates A $(1,3)$, B $(2,2)$, C $(0,3)$. Which candidate is ranked **second**? → `ex1_second` (a string). **Exercise 2 (RotatE composition).** RotatE relation $r_k$ is a vector of angles $\boldsymbol\theta_k$. Write `compose(theta1, theta2)` returning the angles of $r_3$ with $r_3(x)=r_2(r_1(x))$, wrapped to $[0, 2\pi)$. **Exercise 3 (filtered rank).** The true tail scores 0.6; the other candidates score 0.9 (*also a known true tail*), 0.7 and 0.2. Raw rank is 3. What is the **filtered** rank? → `ex3_rank`. """) mo.vstack([_rev, _post])
ex1_second = None · 5 lines
ex1_second = None ex3_rank = None def compose(theta1, theta2): return None
def _verdict(ok, answered): · 49 lines
def _verdict(ok, answered): if not answered: return mo.callout(mo.md("Not answered yet."), kind="neutral") return mo.callout( mo.md("Correct!" if ok else "Not yet — try again or open a hint."), kind="success" if ok else "warn", ) def _check_compose(): try: g = np.random.default_rng(3) t1, t2 = g.uniform(0, 2 * np.pi, 5), g.uniform(0, 2 * np.pi, 5) got = compose(t1, t2) if got is None: return _verdict(False, False) h = g.normal(size=5) + 1j * g.normal(size=5) ok = np.allclose( h * np.exp(1j * t1) * np.exp(1j * t2), h * np.exp(1j * np.asarray(got)), ) and np.all( (np.asarray(got) >= 0) & (np.asarray(got) < 2 * np.pi) ) return _verdict(ok, True) except Exception as e: return mo.callout(mo.md(f"Error: `{e}`"), kind="danger") mo.vstack( [ mo.md("**Exercise 1**"), _verdict(ex1_second == "C", ex1_second is not None), mo.md("**Exercise 2**"), _check_compose(), mo.md("**Exercise 3**"), _verdict(ex3_rank == 2, ex3_rank is not None), mo.accordion( { "Solution 1": mo.md( r"$\mathbf h+\mathbf r=(1,3)$. Distances: A 0, C 1, B $\sqrt2\approx1.41$ → order A, **C**, B." ), "Solution 2": mo.md( "```python\ndef compose(theta1, theta2):\n return np.mod(np.asarray(theta1) + np.asarray(theta2), 2 * np.pi)\n```\nRotations compose by adding angles; this is why RotatE models composition (grandparent = parent ∘ parent: θ_gp = 2 θ_p)." ), "Solution 3": mo.md( "Remove the other known true tail (0.9): only 0.7 beats 0.6 → filtered rank **2** (raw rank 3)." ), } ), ] )
5 · Pitfall: raw (unfiltered) ranking punishes correct answers
A person has several children. When the test triple is (Ana, parent_of, Bo), the model ranking Ana's other child Cy first is right, yet raw ranking counts it as an error. Same trained ComplEx, two protocols:
def mrr(model, rel, filtered): · 27 lines
def mrr(model, rel, filtered): rr, all_t = [], torch.arange(kg_n_entities) with torch.no_grad(): for h, _, t in kg_test[kg_test[:, 1] == rel]: s = model( torch.full((kg_n_entities,), int(h)), torch.full((kg_n_entities,), rel), all_t, ) if filtered: s[[x for x in kg_known[(h, rel)] if x != t]] = -float( "inf" ) rr.append(1.0 / (1 + (s > s[t]).sum().item())) return np.mean(rr) _m = train_kge(ComplEx) pd.DataFrame( [ { "relation": KG_RELATIONS[r], "raw MRR": mrr(_m, r, False), "filtered MRR": mrr(_m, r, True), } for r in range(5) ] ).round(3)
Raw MRR understates every 1-to-N and N-to-N relation (several true tails compete for rank 1), while spouse (1-to-1) is unaffected. Fix: always report filtered metrics for KG completion, and say so. OGB's link-prediction benchmarks avoid the issue by fixing a set of sampled negatives per test edge.
6 · Connecting to your research (SNA on X/Twitter)
-
A social KG has the same patterns: follows is antisymmetric (and 1-to-N), mutual follow / co-retweet are symmetric, retweeted_by is the inverse of retweets, and "retweets someone who retweets X" is a composition. A model choice follows from that list: TransE will fail on the symmetric ones, DistMult on the directional ones; ComplEx / RotatE are safe defaults.
-
KG completion is a fair framing for "predict missing account attributes": (user, located_in, ?), (user, supports, ?).
Summary
- KGE = entity + relation embeddings + a score; train with negative sampling; evaluate by filtered ranking.
- TransE translates, DistMult multiplies (always symmetric), ComplEx adds conjugation (asymmetry), RotatE rotates (inverse, composition, symmetry).
- On the family KG the measured pattern matches the theory: TransE fails symmetric relations (≈ 0.45), DistMult confuses directions (≈ 0.6–0.7), ComplEx and RotatE are best.
- Use filtered metrics.
Next — L11 · GNNs for recommender systems: BPR, LightGCN, PinSage.