Are the best GNN design choices on Cora also the best for bot detection?

Links: CS224W L19 (the GNN design space, task similarity by Kendall τ; You, Ying & Leskovec 2020), L06 (sum vs mean aggregation), notebooks/cs224w/L19_conclusion_design_space.py (the same design space on Cora, CiteSeer and SBMs), experiments/rege-vs-wl (sum beats mean and max for rings). Atlas: GNN design; benchmarks.

Question

L19's answer to "which GNN should I use?" is: find a known task whose design ranking resembles yours and reuse its best design. Benchmarks are mostly homophilous citation graphs. Is bot detection on a retweet graph such a task, or does the benchmark's ranking mislead? And does adding sum aggregation (GIN), which the rege-vs-wl study favours for rings, change the picture?

Hypotheses (written before running)

1. Low similarity. Kendall τ between design rankings on Cora and on bot detection (reciprocal rings) is below 0.3, while τ(Cora, CiteSeer) is above 0.6 (L19 found 0.84 on a smaller sample).

2. Disjoint winners. None of Cora's top-5 designs is among the top-5 for bot detection (reciprocal rings).

3. Sum aggregation for rings. GIN is the best layer type on both bot tasks (mean over the other design dimensions), but not on Cora.

4. Transfer regret. Using Cora's best design for bot detection (reciprocal) costs at least 0.05 AUC against the best design in the space.

Setup

• Design space: L19's 72 designs (layer {GCN, GraphSAGE, GAT} × layers {1, 2, 3} × BatchNorm × skip × hidden {32, 64}) plus the same 24 with a GIN layer: 96 designs. L19's training recipe (pre-MLP, linear head, Adam 0.01, 100 epochs, model selection on validation). Three seeds per design.

• Tasks: Cora, CiteSeer (test accuracy, public split); bot detection on reciprocal and on co-amplification worlds (worlds 0 and 1, 7 standardised account features on the undirected retweet graph, stratified 40/20/40 split, class-weighted loss, test AUC).

• Ranking similarity: Kendall τ over all 96 designs; layer-type effects as marginal means over the full factorial.

• Noise ceiling: τ between the rankings from different seeds of the same task. A cross-task τ can only be read against it (a task whose designs all score about the same has a low ceiling, and a low cross-task τ means little there).

Run: uv run marimo edit experiments/design-transfer/design_transfer.py and press the run button (about 15 minutes with 4 workers; results are cached per design, task and seed in runs.json).

Results (summary; full tables in results.md)

• H1 supported, but read it with the ceiling: the reciprocal task is solved by almost every design (median 0.995), so its ranking is mostly noise. The co-amplification task has a ceiling as high as Cora's and still ranks designs slightly opposite to Cora (τ = −0.21). The two bot tasks are not like each other either (τ = 0.09).

• H2 supported. No overlap between Cora's top 5 and either bot task's top 5.

• H3 refuted. GIN is the best layer on average only for co-amplification rings, by +0.006, and the worst on reciprocal rings. But 4 of the co-amplification top 5 designs are GIN (1 layer, BatchNorm and/or skip).

• H4 refuted for reciprocal rings (regret 0.001: nothing to lose at the ceiling). On co-amplification rings Cora's best design (GCN-2L-64) loses 0.113 AUC.

• Choices that flip between Cora and co-amplification detection: BatchNorm (−0.066 on Cora, +0.073 on bots) and depth (Cora prefers 3 layers, bots 1).

Real-data extension: social networks (added after the first run; hypotheses written before running it)

The first run compared Cora with synthetic bot tasks. Is the gap between Cora and a social task about bots, or about social graphs? Four real social node-classification tasks from gl.datasets, two homophilous (Facebook page-page, page type, adjusted homophily 0.82; LastFM Asia, country, 0.86) and two heterophilous (Twitch England, mature streamer, 0.09; Deezer Europe, gender, 0.03), go through the same 96 designs, recipe and seeds. Binary tasks report AUC, multi-class ones accuracy; split 40/20/40 stratified, fixed across seeds.

5. Homophilous social graphs behave like Cora. τ(Cora, Facebook pages) and τ(Cora, LastFM Asia) are both at least 0.5.
6. Heterophilous social graphs do not. τ(Cora, Twitch-EN) and τ(Cora, Deezer) are both below 0.3.
7. Homophily, not "social", decides transfer. Both homophilous tasks have a higher τ with Cora than both heterophilous tasks.

Run: the second button in the notebook, or uv run python experiments/design-transfer/designs.py --social --workers 10 (about 30 minutes with 10 workers; Facebook pages and Deezer are the slow ones).

Social-network results (summary; full table in results.md)

• H5 refuted. Even the homophilous social graphs rank designs only weakly like Cora (τ = 0.32 and 0.39), although their rankings are stable (seed-to-seed τ ≈ 0.8) and CiteSeer reaches 0.69. Homophily makes the task easy for every message-passing design, not the ranking the same: both prefer attention (GAT) and depth more than Cora does.

• H6 supported. Twitch and Deezer: τ = 0.08 and 0.10 with Cora.

• H7 supported. Both homophilous tasks are closer to Cora than both heterophilous ones, so homophily does order transfer, just from a lower level than expected.

• Regret is small where it matters least. Cora's best design (GCN-2L-64) is within 0.008 of the best on three tasks, because their score spreads are narrow; on Deezer, the one task with a real spread and a high ceiling, it loses 0.052. The same pattern as the co-amplification bots: transfer fails where design choices actually matter.

• Twitch's top three designs are all GIN with BatchNorm (1–2 layers), the family that won co-amplification bot detection; τ(Twitch, bots-co-amplify) is still only +0.03, so this is a shared winner, not a shared ranking.

Next steps

1. Directed layers (separate in/out aggregation), which rege-vs-wl found decisive, as a design dimension.
2. Ring-held-out splits for the bot tasks (honest-splits showed random splits flatter graph models).
3. Anchor-based transfer as in L19: rank 12 anchor designs on a new bot task and pick the nearest known task.

In the repository: experiments/design-transfer/ · README.md config.yaml design_transfer.py designs.py results.md runs.json social_homophily.json