This is a read-only view of the notebook source: text and code, no outputs. To run it, open the live marimo server when the workstation tunnel is up, or run uv run marimo edit notebooks/cs224w/L17_agents_graphs.py.

import warnings

warnings.filterwarnings("ignore", category=FutureWarning)
# harmless Intel-GPU probe message printed by torch on this machine
warnings.filterwarnings("ignore", message=".*Level Zero.*")

import hashlib
import json
import re
import threading
import time
import urllib.request
from concurrent.futures import ThreadPoolExecutor

import marimo as mo
from gl.nb import reveal
import numpy as np
import pandas as pd
import networkx as nx
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.feature_extraction.text import TfidfVectorizer

SEED = 0
rng = np.random.default_rng(SEED)
torch.manual_seed(SEED)
# several open notebooks share the CPU; more threads = contention
torch.set_num_threads(4)
# shared datasets (Cora etc. are cached here) and this folder's disk cache
DATA_DIR = mo.notebook_dir().parents[1] / "data"
CACHE_DIR = mo.notebook_dir() / "cache"
CACHE_DIR.mkdir(exist_ok=True)
plt.rcParams.update(
    {
        "figure.dpi": 100,
        "axes.spines.top": False,
        "axes.spines.right": False,
    }
)

L17 · Agents + graphs — building intelligent and collaborative agents

CS224W Machine Learning with Graphs · hands-on companion to Lecture 17 (guest lecture, Shirley Wu)

Audience: you did L11 (BPR) and L16 (GraphRAG, the ask_llm helper). Prerequisites: none beyond those; an LLM server is used live (team Ollama via reverse proxy; answers cached).

By the end you can:

show that an RLHF reward model's loss is BPR, and train one on preference pairs;
build a small semi-structured knowledge base (product texts + a KG) and see why text-only and KG-only retrieval both fail on STaRK-style queries;
run an AvaTaR-style optimisation loop: a comparator LLM studies the agent's successes vs. failures and rewrites its tool-use instructions (here: how to weight the tools), improving Hit@1;
estimate a multiturn-aware reward (CollabLLM) by simulating future turns with an LLM user simulator, and see why a clarifying question can beat a full answer.

In short. LLMs are pretrained, instruction-tuned and aligned with RLHF, whose reward model is trained with the same pairwise loss as BPR. Real queries mix text and relations over semi-structured KBs (STaRK: every method below 18% Hit@1). AvaTaR optimises an agent's tool-use instructions by contrastive reasoning over good and bad queries. CollabLLM rewards each response by its long-term effect on the conversation, estimated with simulated future turns.

Outline

RLHF's reward model is BPR — train one on synthetic preference pairs.
A semi-structured knowledge base — product texts + a KG; text-only vs. KG-only vs. hybrid retrieval.
AvaTaR-lite — an LLM comparator optimises the agent's tool weights by contrasting good and bad queries.
CollabLLM-lite — multiturn-aware reward with an LLM user simulator.
Exercises, research links.

1 · RLHF: the reward model is trained with BPR

Humans compare two responses $y^+\succ y^-$; the reward model $r_\phi$ is trained with $$\mathcal L=-\log\sigma\big(r_\phi(x,y^+)-r_\phi(x,y^-)\big),$$ exactly BPR from L11 with "user" → prompt and "item" → response. Synthetic check: responses have 5 features (helpfulness, correctness, verbosity, politeness, hedging); hidden human taste $w^\star$; 2,000 noisy comparisons. Can the loss recover $w^\star$?

RESPONSE_FEATURES = [
    "helpfulness",
    "correctness",
    "verbosity",
    "politeness",
    "hedging",
]
w_star = torch.tensor(
    [1.5, 2.0, -0.5, 0.3, -0.8]
)  # hidden human preferences

def preference_pairs(n=2000, seed=SEED):
    g = torch.Generator().manual_seed(seed)
    a, b = torch.randn(n, 5, generator=g), torch.randn(n, 5, generator=g)
    p_a = torch.sigmoid(a @ w_star - b @ w_star)  # Bradley–Terry: P(a ≻ b)
    a_wins = torch.rand(n, generator=g) < p_a
    return torch.where(a_wins[:, None], a, b), torch.where(
        a_wins[:, None], b, a
    )  # (chosen, rejected)

def bpr_loss(pos_score, neg_score):  # the L11 loss, verbatim
    return -F.logsigmoid(pos_score - neg_score).mean()

chosen, rejected = preference_pairs()
reward_w = torch.zeros(5, requires_grad=True)
_opt = torch.optim.Adam([reward_w], lr=0.05)
for _ in range(500):
    _opt.zero_grad()
    bpr_loss(chosen @ reward_w, rejected @ reward_w).backward()
    _opt.step()
_acc = ((chosen @ reward_w) > (rejected @ reward_w)).float().mean().item()
pd.DataFrame(
    {
        "feature": RESPONSE_FEATURES,
        "hidden w*": w_star.numpy(),
        "learned reward weight": reward_w.detach().numpy().round(2),
    }
).assign(note=["", "", "", "", f"pairwise accuracy {_acc:.2f}"])

The reward model recovers the hidden taste: same signs and close magnitudes (Bradley–Terry is identifiable up to scale; with 2,000 noisy pairs the scale is slightly shrunk). Its pairwise accuracy is capped by the label noise humans add. RLHF then optimises the LLM against this $r_\phi$ (PPO, or DPO without an explicit reward model). Note what it does not see: the rest of the conversation. §4 returns to that.

2 · A semi-structured knowledge base (STaRK-lite)

STaRK combines text (product descriptions, abstracts) with relations (brand, category, compatibility, citations). Queries need both. Our synthetic store: 240 toy products, each with a text description (category word + qualities) and KG relations made_by → brand and, for helmets, compatible_with → a scooter/tricycle line. The brand is only in the KG, not in the text.

BRANDS = [ · 91 lines

BRANDS = [
    "Radio Flyer",
    "Little Tikes",
    "Hape",
    "Micro",
    "Strider",
    "Lego",
    "Melissa",
    "Brio",
]
CATEGORIES = [
    "tricycle",
    "scooter",
    "helmet",
    "wagon",
    "kite",
    "puzzle",
    "blocks",
    "doll",
]
QUALITIES = [
    "fun",
    "safe",
    "durable",
    "foldable",
    "lightweight",
    "colorful",
    "quiet",
    "sturdy",
    "classic",
    "compact",
]

def make_store(n=240, seed=SEED):
    r = np.random.default_rng(seed)
    kg = nx.MultiDiGraph()
    rows = []
    for i in range(n):
        cat, brand = r.choice(CATEGORIES), r.choice(BRANDS)
        quals = list(r.choice(QUALITIES, 3, replace=False))
        text = f"A {quals[0]}, {quals[1]} {cat} for kids. Designed to be {quals[2]}."
        rows.append(
            {
                "id": i,
                "category": cat,
                "brand": brand,
                "qualities": quals,
                "text": text,
            }
        )
        kg.add_edge(f"p{i}", brand, rel="made_by")
        kg.add_edge(f"p{i}", cat, rel="is_a")
    products = pd.DataFrame(rows)
    rides = products[
        products.category.isin(["scooter", "tricycle"])
    ].id.tolist()
    for i in products[products.category == "helmet"].id:
        kg.add_edge(
            f"p{i}", f"p{int(r.choice(rides))}", rel="compatible_with"
        )
    return products, kg

store, store_kg = make_store()

def make_queries(n=120, seed=SEED + 1):
    """Each query names a brand (KG), a category (text) and two qualities (text); some ask for a compatible helmet."""
    r = np.random.default_rng(seed)
    qs = []
    for _ in range(n):
        p = store.iloc[int(r.integers(len(store)))]
        q1, q2 = r.choice(p.qualities, 2, replace=False)
        if p.category == "helmet" and r.random() < 0.6:
            ride = next(
                v
                for _, v, d in store_kg.out_edges(f"p{p.id}", data=True)
                if d["rel"] == "compatible_with"
            )
            ride_row = store.iloc[int(ride[1:])]
            text = f"a {q1} helmet from {p.brand} that fits the {ride_row.brand} {ride_row.category}"
        else:
            text = f"a {q1} and {q2} {p.category} from {p.brand}"
        qs.append({"query": text, "gold": int(p.id)})
    return pd.DataFrame(qs)

store_queries = make_queries()
mo.vstack(
    [
        store[["id", "category", "brand", "text"]].head(4),
        store_queries.head(4),
    ]
)

_tfidf = TfidfVectorizer().fit( · 73 lines

_tfidf = TfidfVectorizer().fit(
    list(store.text) + list(store_queries["query"])
)
_text_matrix = _tfidf.transform(store.text)
_brand_of = dict(zip(store.id, store.brand))
_compat = {
    int(u[1:]): int(v[1:])
    for u, v, d in store_kg.edges(data=True)
    if d["rel"] == "compatible_with"
}

def parse_query(q):
    """Light query parsing: product brand (after 'from'), category word, and the ride a helmet must fit."""
    brand = next((b for b in BRANDS if f"from {b}" in q), None)
    cat = next(
        (
            c
            for c in CATEGORIES
            if re.search(rf"\b{c}\b", q.split(" from ")[0])
        ),
        None,
    )
    fits = re.search(r"fits the (.+) (\w+)$", q)
    return brand, cat, (fits.group(1), fits.group(2)) if fits else None

def tool_scores(q):
    """[n_products, 4] matrix of tool outputs: text similarity, brand match (KG), category match, compatibility (KG)."""
    brand, cat, fits = parse_query(q)
    text = (_text_matrix @ _tfidf.transform([q]).T).toarray().ravel()
    b = np.array([float(_brand_of[i] == brand) for i in store.id])
    c = np.array([float(x == cat) for x in store.category])
    comp = np.zeros(len(store))
    if fits:
        for h, ride in _compat.items():
            comp[h] = float(
                store.brand[ride] == fits[0]
                and store.category[ride] == fits[1]
            )
    return np.stack([text, b, c, comp], 1)

TOOLS = [
    "text_similarity",
    "brand_match_kg",
    "category_match",
    "compatible_with_kg",
]
store_tool_scores = [tool_scores(q) for q in store_queries["query"]]

def hit_at_1(weights, idx):
    """Expected Hit@1 with random tie-breaking."""
    hits = []
    for i in idx:
        s = store_tool_scores[i] @ np.asarray(weights, dtype=float)
        top = np.flatnonzero(np.isclose(s, s.max()))
        hits.append(float(store_queries.gold[i] in top) / len(top))
    return float(np.mean(hits))

TRAIN_Q, VAL_Q = range(0, 80), range(80, 120)
pd.DataFrame(
    [
        {
            "retriever": n,
            "weights (text, brand, category, compat)": w,
            "Hit@1 train": hit_at_1(w, TRAIN_Q),
            "Hit@1 val": hit_at_1(w, VAL_Q),
        }
        for n, w in [
            ("text only (vector search)", [1, 0, 0, 0]),
            ("KG only (brand + category)", [0, 1, 1, 0]),
            ("all tools, equal weights", [1, 1, 1, 1]),
        ]
    ]
).round(3)

Text-only retrieval (vector search) gets Hit@1 ≈ 0.40: it cannot see the brand, which lives only in the KG. KG-only gets ≈ 0.24: brand + category leave many ties, and the qualities are only in the text. Using all tools reaches ≈ 0.89 / 0.94. That is STaRK's lesson in miniature: real queries are semi-structured, so a retriever must combine textual and relational evidence. The remaining errors are genuine ties (several products match every clue).

3 · AvaTaR-lite: optimise the agent's instructions by contrastive reasoning

The "agent" scores products with $\sum_k w_k\cdot\text{tool}_k$; its instructions are the weights. AvaTaR's loop:

run the agent on training queries; collect successes and failures;
show a comparator LLM the current instructions plus, for each failure, the tool scores of the correct product vs. the product the agent ranked first (and a few successes for contrast);
the comparator writes improved instructions (here: new weights, as JSON); repeat; keep the best on validation.

We start from the naive "text-only" agent.

import os · 90 lines

import os

# Point GL_OLLAMA_URL at your Ollama server (e.g. the team reverse proxy); defaults to a local Ollama.
OLLAMA_URL = os.environ.get("GL_OLLAMA_URL", "http://localhost:11434")
LLM_CACHE = CACHE_DIR / "llm_cache.json"
_cache_lock = threading.Lock()

def _load_cache():
    try:
        return json.loads(LLM_CACHE.read_text())
    except (FileNotFoundError, json.JSONDecodeError):
        return {}

def ask_llm(prompt, model="gemma4:latest", timeout=300, temperature=0.0):
    """One chat call to the team Ollama server, cached on disk by (model, temperature, prompt). Thread-safe."""
    key = hashlib.sha1(
        f"{model}\n{temperature}\n{prompt}".encode()
    ).hexdigest()
    with _cache_lock:
        cached = _load_cache().get(key)
    if cached is not None:
        return cached
    body = json.dumps(
        {
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": False,
            "think": False,
            "options": {"temperature": temperature},
        }
    ).encode()
    try:
        req = urllib.request.Request(
            f"{OLLAMA_URL}/api/chat",
            data=body,
            headers={"Content-Type": "application/json"},
        )
        answer = json.loads(
            urllib.request.urlopen(req, timeout=timeout).read()
        )["message"]["content"].strip()
    except Exception as e:
        return f"[LLM error: {e}]"
    with _cache_lock:
        cache = _load_cache()
        cache[key] = answer
        tmp = LLM_CACHE.with_suffix(".tmp")
        tmp.write_text(json.dumps(cache))
        tmp.replace(LLM_CACHE)
    return answer

def describe_case(i, weights):
    s = store_tool_scores[i] @ np.asarray(weights, float)
    top = int(np.argmax(s))
    gold = int(store_queries.gold[i])
    fmt = lambda p: ", ".join(
        f"{t}={store_tool_scores[i][p, k]:.2f}"
        for k, t in enumerate(TOOLS)
    )
    return f'Query: "{store_queries["query"][i]}"\n  correct product: {fmt(gold)}\n  agent ranked first: {fmt(top)}'

def comparator_prompt(weights, failures, successes):
    w = {t: round(float(x), 3) for t, x in zip(TOOLS, weights)}
    return (
        "You are optimizing a product-search agent. It scores every product as a weighted sum of tool outputs "
        f"(each tool output is between 0 and 1). Current weights: {json.dumps(w)}.\n\n"
        "Queries where the agent FAILED (it ranked a wrong product first):\n"
        + "\n".join(failures)
        + "\n\nQueries where it SUCCEEDED:\n"
        + "\n".join(successes)
        + "\n\nCompare the failures with the successes. Which tools separate the correct product from the wrong one? "
        "Return improved weights as JSON with exactly these keys: "
        + ", ".join(TOOLS)
        + ". Weights must be between 0 and 1. Reply with one line of reasoning, then the JSON object."
    )

def parse_weights(text, fallback):
    m = re.search(r"\{[^{}]*\}", text, re.S)
    try:
        d = json.loads(m.group(0))
        return [float(np.clip(float(d[t]), 0, 1)) for t in TOOLS]
    except Exception:
        return list(fallback)

avatar_model = mo.ui.dropdown(
    ["gemma4:latest", "gemma4:26b", "qwen3.6:27b"],
    value="gemma4:latest",
    label="comparator LLM",
)
avatar_run = mo.ui.run_button(label="Run 4 optimisation rounds (cached)")
mo.hstack([avatar_model, avatar_run], justify="start")

mo.stop( · 55 lines

mo.stop(
    not avatar_run.value,
    mo.md("Press the button to run the contrastive-reasoning loop."),
)
_w = [1.0, 0.0, 0.0, 0.0]
_r = np.random.default_rng(SEED)
avatar_log = [
    {
        "round": 0,
        "weights": [round(x, 2) for x in _w],
        "Hit@1 train": hit_at_1(_w, TRAIN_Q),
        "Hit@1 val": hit_at_1(_w, VAL_Q),
        "comparator says": "(start: text-only)",
    }
]
for _round in range(1, 5):
    _res = [
        (
            i,
            int(np.argmax(store_tool_scores[i] @ np.asarray(_w)))
            == store_queries.gold[i],
        )
        for i in TRAIN_Q
    ]
    _fail = [i for i, ok in _res if not ok]
    _succ = [i for i, ok in _res if ok]
    _f = (
        [
            describe_case(i, _w)
            for i in _r.choice(_fail, min(6, len(_fail)), replace=False)
        ]
        if _fail
        else []
    )
    _s = (
        [
            describe_case(i, _w)
            for i in _r.choice(_succ, min(3, len(_succ)), replace=False)
        ]
        if _succ
        else []
    )
    _reply = ask_llm(comparator_prompt(_w, _f, _s), avatar_model.value)
    _w = parse_weights(_reply, _w)
    avatar_log.append(
        {
            "round": _round,
            "weights": [round(x, 2) for x in _w],
            "Hit@1 train": hit_at_1(_w, TRAIN_Q),
            "Hit@1 val": hit_at_1(_w, VAL_Q),
            "comparator says": _reply.split("\n")[0][:160],
        }
    )
avatar_log = pd.DataFrame(avatar_log)
avatar_log.round(3)

What the LLM did (gemma4, cached): from the text-only agent (Hit@1 0.40), round 1 noticed that failures were decided by tiny text-similarity differences while the brand tool separated correct from wrong products, and moved weight to brand and category → 0.84 (val 0.94). Round 2 added the compatibility tool after studying a helmet failure → 0.89, the best achievable here (the rest are ties). Later rounds just fine-tune. That is AvaTaR's contrastive reasoning: compare what works with what doesn't, and write actionable instructions. The real AvaTaR edits natural-language tool-use instructions and code, not just four weights.

4 · CollabLLM-lite: rewarding long-term collaboration

Single-turn RLHF rewards the response that looks best now, so a full draft beats a clarifying question. CollabLLM instead estimates the multiturn-aware reward: simulate the conversation's future with an LLM user simulator that knows the user's hidden intent, then score how the conversation ends (task quality, minus a cost for tokens the user must read).

Five requests (four ambiguous, one not), two policies: answer now vs. ask one clarifying question first. Each conversation: assistant turn → simulated user reply → assistant's final draft → an LLM judge scores the draft against the hidden intent (1–10).

COLLAB_TASKS = [ · 89 lines

COLLAB_TASKS = [
    {
        "request": "I need to write about how optimism can improve our well-being.",
        "intent": "one short, warm, inspirational paragraph (under 80 words) focused on relationships with friends and family",
    },
    {
        "request": "Give me a good pasta recipe.",
        "intent": "a vegetarian, nut-free pasta recipe ready in 15 minutes, for two people",
    },
    {
        "request": "Write a tweet announcing our new study.",
        "intent": "a neutral, academic-sounding tweet about a study of polarisation on X/Twitter in Iran, with no emojis and no hashtags",
    },
    {
        "request": "Explain graph neural networks.",
        "intent": "a two-sentence explanation for a 12-year-old, using an analogy about friends sharing news",
    },
    {
        "request": "Fix the spelling in this sentence: 'Teh cat sat on teh mat.'",
        "intent": "just the corrected sentence, nothing else",
    },
]
ASSISTANT_LLM, JUDGE_LLM = "gemma4:latest", "gemma4:26b"

def assistant_turn(history, policy):
    if policy == "ask first":
        rule = "Reply with exactly ONE short clarifying question (at most 20 words) about what the user wants. Do not answer yet."
    else:
        rule = "Answer the request directly and completely."
    return ask_llm(
        f"You are a helpful assistant. {rule}\n\nConversation so far:\n{history}\nAssistant:",
        ASSISTANT_LLM,
    )

def user_simulator(history, intent):
    return ask_llm(
        "You are simulating a human user talking to an AI assistant. Your hidden goal: "
        + intent
        + ". Reply briefly (1–2 sentences), as the user, to the assistant's last message. If it asked a question, answer it "
        "according to your goal; if it gave an answer that misses your goal, say precisely what to change.\n\n"
        f"Conversation so far:\n{history}\nUser:",
        ASSISTANT_LLM,
    )

def final_draft(history):
    return ask_llm(
        "You are a helpful assistant. Based on the whole conversation, write the complete final response to the user's "
        "original request, following every preference they expressed. Output only that response.\n\n"
        f"{history}\nFinal response:",
        ASSISTANT_LLM,
    )

def judge(text, goal):
    reply = ask_llm(
        f"Rate from 1 (useless) to 10 (perfect) how well the RESPONSE fulfils this GOAL.\nGOAL: {goal}\n"
        f"RESPONSE:\n{text}\n\nAnswer with a single integer.",
        JUDGE_LLM,
    )
    m = re.search(r"\d+", reply)
    return min(10, int(m.group())) if m else 0

def simulate(task, policy):
    h = f"User: {task['request']}"
    first = assistant_turn(h, policy)
    single_turn = judge(
        first, task["request"]
    )  # single-turn view: judged against the request only
    h += f"\nAssistant: {first}"
    h += f"\nUser: {user_simulator(h, task['intent'])}"
    final = final_draft(h)
    words = len(first.split()) + len(final.split())
    quality = judge(
        final, task["intent"]
    )  # multiturn view: the conversation's outcome
    return {
        "request": task["request"][:45],
        "policy": policy,
        "single-turn reward": single_turn,
        "final quality (1–10)": quality,
        "words the user read": words,
        "multiturn reward": quality - 0.005 * words,
        "first response": first[:150],
        "final": final[:200],
    }

collab_run = mo.ui.run_button(
    label="Simulate 10 conversations (≈ 1–2 min uncached)"
)
collab_run

mo.stop(
    not collab_run.value,
    mo.md(
        "Press the button to simulate both policies on the five requests."
    ),
)
_jobs = [(t, p) for t in COLLAB_TASKS for p in ["answer now", "ask first"]]
with ThreadPoolExecutor(5) as _pool:
    collab_results = pd.DataFrame(
        list(_pool.map(lambda j: simulate(*j), _jobs))
    )
mo.vstack(
    [
        collab_results.groupby("policy")[
            [
                "single-turn reward",
                "final quality (1–10)",
                "words the user read",
                "multiturn reward",
            ]
        ]
        .mean()
        .round(2),
        collab_results.drop(columns=["first response", "final"]).round(2),
        mo.accordion(
            {
                "first responses": collab_results[
                    ["request", "policy", "first response"]
                ]
            }
        ),
    ]
)

_rev = reveal( · 34 lines

_rev = reveal(
    r"""
Measured (gemma4 assistant and user simulator, gemma4:26b judge, one simulated future per conversation, cached):

| policy | single-turn reward | final quality (1–10) | words the user read | multiturn reward |
|---|---|---|---|---|
| answer now | **10.0** | 4.8 | 929 | 0.2 |
| ask one question first | 1.4 | 4.6 | **269** | **3.3** |

* A **single-turn** reward model loves the immediate full answer (10 vs. 1.4). That is the bias CollabLLM targets.
* After the simulated future, final quality is about **equal**, but asking first spares the user ≈ 70% of the reading, so the
  **multiturn-aware reward** flips the preference (3.3 vs. 0.2). On the pasta request the question also led to a better final answer (reward 8.1 vs 5.6).
* For the **unambiguous** spelling fix, asking adds nothing (equal quality, slightly more words). A model trained on multiturn rewards
  learns *when* to ask, which is what the paper found (≈ 3× more questions on ambiguous queries, only +18% on clear ones).
* Caveat (the lecture's too): a single simulated future per conversation, and an 8B simulator, are noisy estimates. CollabLLM samples
  several futures and validated with 201 real users.

""",
    predict="Which policy wins under a single-turn reward model, and which under a multiturn-aware reward? Does asking first change final quality?",
)
_post = mo.md(r"""
---

## 5 · Exercises

**Exercise 1 (reward model = BPR).** A reward model scores the chosen response 2.0 and the rejected one 0.5. Loss? → `ex1_loss` (3 decimals).

**Exercise 2 (tool weights by hand).** Find weights `my_weights` (text, brand, category, compat) with **text weight ≤ 0.2** that reach
Hit@1 ≥ 0.9 on the validation queries.

**Exercise 3 (multiturn reward).** A conversation ends with quality 7 and the user read 600 words. With penalty 0.005 per word,
what is the multiturn reward? → `ex3_reward`.
""")
mo.vstack([_rev, _post])

ex1_loss = None · 3 lines

ex1_loss = None
my_weights = None  # e.g. [0.1, 0.5, 0.3, 0.2]
ex3_reward = None

def _verdict(ok, answered): · 51 lines

def _verdict(ok, answered):
    if not answered:
        return mo.callout(mo.md("Not answered yet."), kind="neutral")
    return mo.callout(
        mo.md("Correct!" if ok else "Not yet — try again or open a hint."),
        kind="success" if ok else "warn",
    )

_w_ok = (
    my_weights is not None
    and len(my_weights) == 4
    and my_weights[0] <= 0.2
    and hit_at_1(my_weights, VAL_Q) >= 0.9
)
mo.vstack(
    [
        mo.md("**Exercise 1**"),
        _verdict(
            ex1_loss is not None
            and abs(ex1_loss - float(-F.logsigmoid(torch.tensor(1.5))))
            < 2e-3,
            ex1_loss is not None,
        ),
        mo.md(
            "**Exercise 2**"
            + (
                f" — your weights reach val Hit@1 = {hit_at_1(my_weights, VAL_Q):.3f}"
                if my_weights is not None and len(my_weights) == 4
                else ""
            )
        ),
        _verdict(_w_ok, my_weights is not None),
        mo.md("**Exercise 3**"),
        _verdict(
            ex3_reward is not None and abs(ex3_reward - 4.0) < 1e-6,
            ex3_reward is not None,
        ),
        mo.accordion(
            {
                "Solution 1": mo.md(
                    r"$-\log\sigma(2.0-0.5)=-\log\sigma(1.5)\approx 0.201$, exactly the BPR loss of L11."
                ),
                "Solution 2": mo.md(
                    "For example `[0.1, 0.5, 0.3, 0.2]`: text only breaks ties among products that already match brand and category; "
                    "the KG tools must dominate. Any weights where brand + category + compat outweigh the small text differences work."
                ),
                "Solution 3": mo.md(r"$7-0.005\cdot600=7-3=$ **4.0**."),
            }
        ),
    ]
)

6 · Connecting to your research (SNA on X/Twitter)

Your tweet store is a semi-structured KB: tweet texts + a graph of users, retweets, hashtags. Analyst questions ("tweets by accounts in cluster 3 that criticise X and were retweeted by Y's followers") need text tools and graph tools. Build them as explicit tools and let an AvaTaR-style loop tune how the agent combines them on a few labelled queries.
Your gemma4 tagging prompts can be optimised the same way: show the comparator LLM tweets it tagged right and wrong (from your audited ~100 rows) and ask for a revised instruction, then re-measure. Cache by prompt hash as you already do.
For analyst-facing assistants, the CollabLLM lesson applies: ask which camp / period / language the user means before running an expensive query.

Summary

RLHF's reward model is trained with the BPR loss; it recovered the hidden preference weights (pairwise accuracy 0.86 under label noise).
Semi-structured retrieval: text-only Hit@1 0.40, KG-only 0.24, combined 0.89.
AvaTaR-lite with a real LLM comparator: 0.40 → 0.84 → 0.89 in two rounds by contrasting failures with successes.
CollabLLM-lite: single-turn reward prefers answering (10 vs 1.4); the multiturn-aware reward prefers asking first on ambiguous requests (3.3 vs 0.2).

Next — L18 · Deep generative models for graphs: GraphRNN and GCPN.