How much does a random split overestimate a bot detector?

Links: CS224W L05 (GNN training: dataset splits, transductive vs inductive, the L05 proof workshop on AUC), notebooks/project/retweet_thread.py §L05 and §L06–07 (the same detector). Atlas: evaluation; social networks.

Question

Graph benchmarks split nodes at random. In a coordination problem that means members of the same ring land on both sides of the split, and anything the model learns about one member (its targets, its partners, its label if the model uses neighbours' labels) transfers to the others for free. A platform never gets that: the campaign it must catch next month is made of accounts it has never labelled. How large is the gap between the number a paper would report and the number a deployment would see?

Hypotheses (written before running)

1. Label-free model. For a detector that uses only features and structure (no labels of neighbours), a random node split and a ring-held-out split agree within 0.03 AUC.

2. Label-aware model. For a detector that also uses training labels of neighbours (share of known-bot neighbours, label propagation), the random split overestimates the ring-held-out split by at least 0.10 AUC.

3. New world = unseen rings. Training on one world and testing on another gives the label-free model about the same AUC as the ring-held-out split (within 0.03).

4. Early detection. Testing on a new world observed only for its first 10 days (of 30) costs the label-free model at least 0.10 AUC against full observation.

5. Link prediction. For Adamic–Adar, common neighbours and preferential attachment, a random edge split overestimates the temporal split (days 0–20 → new edges in days 20–30) by at least 0.05 AUC in every world type.

Setup

• Worlds: reciprocal and co-amplification rings (gl.exp.WORLD_TYPES), seeds in config.yaml.

• Label-free detector: 7 account features, their neighbourhood means, reciprocity and mutual triangles (the project's feature set 4), logistic regression.

• Label-aware detector: the same plus three features built from the training labels only: share of neighbours that are known bots, share that are known humans, and a label-propagation score.

• Protocols: random stratified 5-fold node split; ring-held-out (each fold holds out one whole ring and a random share of humans); cross-world (train on world a, test on world b, where 10% of humans are known and no bot is); early detection (cross-world, test world truncated at day 10).

Run: uv run marimo edit experiments/honest-splits/honest_splits.py and press the run button. The notebook writes results.md and results.json.

Added after the first run (not tests): a features-only model, because the full label-free model was at the ceiling on reciprocal worlds; and a 1-hop-only label-aware model, to locate the leak.

Results (summary; full tables in results.md)

• H1 refuted, narrowly. Label-free gap 0.005 (reciprocal) but 0.031 (co-amplify), just over the 0.03 threshold.

• H2 supported, strongly. The label-aware model loses 0.246 (reciprocal) and 0.176 (co-amplify) when whole rings are held out. Worse: on held-out rings it is below the label-free model (0.754 vs 0.991), because it learned to rely on "my neighbours are known bots", which is zero for a ring nobody has labelled. A model selected on a random split is the wrong one to deploy.

• Where the leak travels. With 1-hop label features only, the gap is 0.190 on reciprocal rings (members are neighbours) but 0.074 on co-amplification rings; label propagation adds the rest there, carrying labels two hops through shared targets.

• H3 refuted. Cross-world is higher than ring-held-out on co-amplification worlds (0.954 vs 0.919): ring-held-out trains on only two of three rings, so it is pessimistic about data size as well as honest about leakage.

• H4 refuted. Ten days instead of thirty cost 0.10 on reciprocal worlds (0.995 → 0.894) but only 0.04 on co-amplification worlds.

• H5 supported. Random edge splits overestimate temporal ones by 0.12–0.14 AUC for all three heuristics in both worlds.

Real-data extension: temporal link prediction (added after the first run; hypotheses written before running it)

Hypothesis 5 held on synthetic worlds. Real networks grow by mechanisms nobody wrote down, so the gap could be larger, smaller or reversed. Five timestamped social networks from gl.datasets: CollegeMsg (messages), EU e-mail (temporal), Bitcoin OTC (trust ratings), MathOverflow (answers and comments) and Higgs Twitter (retweets only). Each pair's first interaction time decides its side of the split: pairs that first appear before the 80% quantile form the past graph, later pairs between two accounts already in the past graph are the temporal positives. The random split hides the same number of pairs drawn from all pairs. Negatives are random non-edges; three seeds; at most 5,000 positives per split.

6. The leak is real. On every real network, the random split overestimates the temporal split by at least 0.05 AUC for common neighbours and for Adamic–Adar.

7. Growth is rich-get-richer. For preferential attachment the gap (random − temporal) is smaller than for Adamic–Adar on every network, because new real ties attach to hubs more than they close triangles.

Run: the second button in the notebook (a few minutes; Higgs and MathOverflow are the large ones). Results go to real_results.json and a separate section of results.md.

Real-data results (summary; full table in results.md)

Random − temporal AUC (means over 3 seeds):

• H6 refuted. The random split overestimates the neighbourhood heuristics on four of five networks, but by ≥ 0.05 only on CollegeMsg (+0.13, as large as in the synthetic world); on e-mail, Bitcoin and MathOverflow the gap is 0.04–0.05. The direction of hypothesis 5 holds on real data; its size was a property of the synthetic world.

• H7 supported. Preferential attachment's gap is smaller everywhere, and negative on Bitcoin, MathOverflow and Higgs: in a temporal split new ties go to hubs, so degree predicts the future better than it predicts a random hold-out.

• Higgs is a different question, not a counter-example. 55% of randomly hidden Higgs pairs touch an account with no other tie (1–10% elsewhere): most retweeters act once, so the random split is mostly cold start (degree 0 after hiding, PA = 0), while the temporal protocol keeps only pairs of accounts already seen. Read the Higgs row as "random splits of cascade data test cold start", which is a protocol choice to state, not a leak.

Next steps

1. A trained GNN with label features (label trick, Wang & Leskovec 2021) under the same protocols.

2. Train-time ring masking: drop the label features of a random half of known bots during training so the model cannot lean on them (the remedy the label trick uses).

3. Temporal campaigns: rings that start mid-month, with a split by account creation time.

In the repository: experiments/honest-splits/ · README.md config.yaml honest_splits.py real_results.json results.json results.md