Validation results¶
Generated by benchmarks/report.py on 2026-10-09 for graphdml 0.1.0. The protocol and pass criteria are in the validation protocol; raw per-seed results are in benchmarks/results/*.csv.
Coverage¶
Stage 1: 300 seeds, 2,000-node graphs, nominal 95% intervals, band 0.95 ± 0.038. se_over_sd is the mean standard error divided by the empirical standard deviation of the estimates (1 = calibrated).
linear, continuous T, oracle nuisances
| graph | effect | coverage | se_over_sd | rmse | bias | n_focal | in_band |
|---|---|---|---|---|---|---|---|
| ER (avg deg 4) | direct | 0.920 | 0.904 | 0.052 | 0.002 | 454.237 | True |
| ER (avg deg 4) | peer | 0.940 | 1.016 | 0.061 | 0.000 | 454.237 | True |
| BA (m=2, hubs) | direct | 0.937 | 0.946 | 0.060 | 0.005 | 310.237 | True |
| BA (m=2, hubs) | peer | 0.927 | 0.913 | 0.090 | 0.007 | 310.237 | True |
| SBM (40 blocks) | direct | 0.947 | 0.965 | 0.068 | 0.000 | 231.577 | True |
| SBM (40 blocks) | peer | 0.970 | 1.044 | 0.109 | -0.009 | 231.577 | True |
linear, continuous T, linear (correct) nuisances
| graph | effect | coverage | se_over_sd | rmse | bias | n_focal | in_band |
|---|---|---|---|---|---|---|---|
| ER (avg deg 4) | direct | 0.923 | 0.915 | 0.052 | 0.002 | 454.237 | True |
| ER (avg deg 4) | peer | 0.930 | 1.012 | 0.061 | 0.001 | 454.237 | True |
| BA (m=2, hubs) | direct | 0.940 | 0.933 | 0.061 | 0.005 | 310.237 | True |
| BA (m=2, hubs) | peer | 0.907 | 0.901 | 0.091 | 0.007 | 310.237 | False |
| SBM (40 blocks) | direct | 0.953 | 0.969 | 0.068 | 0.000 | 231.577 | True |
| SBM (40 blocks) | peer | 0.957 | 1.053 | 0.108 | -0.009 | 231.577 | True |
benchmark linear, binary T, oracle nuisances
| graph | effect | coverage | se_over_sd | rmse | bias | n_focal | in_band |
|---|---|---|---|---|---|---|---|
| ER (avg deg 4) | direct | 0.970 | 1.107 | 0.085 | 0.001 | 454.237 | True |
| ER (avg deg 4) | peer | 0.950 | 0.929 | 0.076 | 0.003 | 454.237 | True |
| BA (m=2, hubs) | direct | 0.950 | 0.993 | 0.116 | 0.005 | 310.237 | True |
| BA (m=2, hubs) | peer | 0.960 | 1.031 | 0.078 | 0.001 | 310.237 | True |
| SBM (40 blocks) | direct | 0.940 | 0.954 | 0.142 | 0.005 | 231.577 | True |
| SBM (40 blocks) | peer | 0.947 | 0.981 | 0.075 | 0.001 | 231.577 | True |
Stage 2 (confirmation of flagged cells): 2000 fresh seeds, band 0.95 ± 0.015.
| dgp | nuisances | graph | effect | coverage | se_over_sd | rmse | bias | n_focal | in_band |
|---|---|---|---|---|---|---|---|---|---|
| linear, continuous T | linear (correct) | BA (m=2, hubs) | direct | 0.939 | 0.994 | 0.058 | 0.001 | 310.327 | True |
| linear, continuous T | linear (correct) | BA (m=2, hubs) | peer | 0.944 | 0.982 | 0.083 | -0.000 | 310.327 | True |
Design choices¶
100 seeds per dataset, default machine-learning nuisances (regularised gradient boosting). Each cell: relative bias · coverage of 95% intervals · RMSE.
| dataset | effect | default (IV-type, buffered, 2 hops) | partialling-out score | focal-only training | 1-hop features | partialling-out + 1-hop | original procedure |
|---|---|---|---|---|---|---|---|
| flu_town (n=3000) | direct | -1.1% · 0.95 · 0.17 | +1.9% · 0.92 · 0.184 | -0.6% · 0.95 · 0.191 | -0.9% · 0.97 · 0.171 | +1.7% · 0.93 · 0.186 | +2.7% · 0.91 · 0.257 |
| flu_town (n=3000) | peer | +0.9% · 0.93 · 0.0867 | +6.0% · 0.97 · 0.0887 | -8.0% · 0.95 · 0.153 | +1.3% · 0.95 · 0.084 | +6.8% · 0.97 · 0.084 | +4.4% · 0.90 · 0.114 |
| referral_app (n=4000) | direct | +2.4% · 0.95 · 0.941 | +0.8% · 0.96 · 0.928 | +10.0% · 0.80 · 1.55 | +2.1% · 0.95 · 0.925 | +0.4% · 0.96 · 0.909 | +13.8% · 0.75 · 2.16 |
| referral_app (n=4000) | peer | +0.5% · 0.93 · 1.63 | -1.9% · 0.93 · 1.94 | +1.7% · 0.89 · 2.51 | +0.6% · 0.91 · 1.67 | -1.6% · 0.91 · 1.94 | -3.6% · 0.84 · 3.33 |
| benchmark linear, ER n=3000 | direct | -0.1% · 0.96 · 0.084 | -0.2% · 0.92 · 0.157 | +0.0% · 0.96 · 0.087 | -0.1% · 0.94 · 0.0814 | -0.1% · 0.94 · 0.14 | -0.4% · 0.95 · 0.37 |
| benchmark linear, ER n=3000 | peer | +0.0% · 0.99 · 0.0591 | -0.5% · 0.94 · 0.102 | +0.2% · 0.90 · 0.0834 | -0.0% · 0.99 · 0.0612 | -0.5% · 0.93 · 0.104 | -3.5% · 0.82 · 0.439 |
| benchmark nonlinear, ER n=3000 | direct | +0.0% · 0.96 · 0.073 | -0.3% · 0.91 · 0.218 | +0.0% · 0.96 · 0.0751 | +0.0% · 0.95 · 0.0733 | -0.1% · 0.96 · 0.182 | -0.4% · 0.92 · 0.387 |
| benchmark nonlinear, ER n=3000 | peer | +0.1% · 0.96 · 0.0503 | -0.4% · 1.00 · 0.11 | +0.2% · 0.95 · 0.0612 | +0.1% · 0.97 · 0.0512 | -0.8% · 0.95 · 0.126 | -3.6% · 0.84 · 0.427 |
Total effect (ATE)¶
Truth = direct + peer × average exposure when all nodes are treated.
| setting | reps | truth | mean_est | rel_bias | coverage | se_over_sd | band |
|---|---|---|---|---|---|---|---|
| ER, mean, oracle | 300 | 1.491 | 1.493 | 0.001 | 0.953 | 0.980 | 0.95 ± 0.038 |
| ER, mean, linear (correct) | 300 | 1.491 | 1.494 | 0.002 | 0.950 | 0.996 | 0.95 ± 0.038 |
| ER, sum, oracle | 300 | 2.995 | 2.994 | -0.000 | 0.957 | 0.978 | 0.95 ± 0.038 |
| BA (hubs), mean, oracle | 300 | 1.500 | 1.512 | 0.008 | 0.920 | 0.913 | 0.95 ± 0.038 |
| BA (hubs), mean, linear (correct) | 300 | 1.500 | 1.512 | 0.008 | 0.913 | 0.900 | 0.95 ± 0.038 |
| BA (hubs), sum, oracle | 300 | 2.998 | 3.016 | 0.006 | 0.923 | 0.906 | 0.95 ± 0.038 |
| flu_town (n=3000), HGB | 100 | -4.933 | -4.930 | 0.001 | 0.940 | 0.988 | 0.95 ± 0.065 |
| referral_app (n=4000), HGB | 100 | 52.000 | 52.478 | 0.009 | 0.900 | 0.941 | 0.95 ± 0.065 |
The cost of the focal set¶
100 seeds, default learners. Using all nodes (no focal set) lowers RMSE substantially, but its i.i.d. variance ignores dependence between neighbors' scores, so for most effects the standard errors are too small (se_over_sd < 1) and coverage falls below nominal. Closing this gap with a dependence-robust variance and cluster cross-fitting is the top V1.1 item.
| dataset | effect | method | rel_bias | rmse | coverage | se_over_sd |
|---|---|---|---|---|---|---|
| flu_town (n=3000) | direct | GraphDML (focal set) | -0.011 | 0.170 | 0.950 | 1.027 |
| flu_town (n=3000) | peer | GraphDML (focal set) | 0.009 | 0.087 | 0.930 | 1.040 |
| flu_town (n=3000) | direct | GraphDML features, all nodes | -0.008 | 0.062 | 0.980 | 1.112 |
| flu_town (n=3000) | peer | GraphDML features, all nodes | 0.005 | 0.031 | 0.880 | 0.866 |
| flu_town (n=3000) | direct | 1-hop aggregates, all nodes | -0.007 | 0.063 | 0.980 | 1.084 |
| flu_town (n=3000) | peer | 1-hop aggregates, all nodes | 0.005 | 0.031 | 0.880 | 0.836 |
| referral_app (n=4000) | direct | GraphDML (focal set) | 0.024 | 0.941 | 0.950 | 1.034 |
| referral_app (n=4000) | peer | GraphDML (focal set) | 0.005 | 1.628 | 0.930 | 1.018 |
| referral_app (n=4000) | direct | GraphDML features, all nodes | 0.014 | 0.364 | 0.900 | 0.893 |
| referral_app (n=4000) | peer | GraphDML features, all nodes | 0.004 | 0.656 | 0.920 | 0.928 |
| referral_app (n=4000) | direct | 1-hop aggregates, all nodes | 0.012 | 0.343 | 0.910 | 0.913 |
| referral_app (n=4000) | peer | 1-hop aggregates, all nodes | 0.003 | 0.643 | 0.920 | 0.933 |
Real network: LastFM Asia (semi-synthetic)¶
100 replications on the real LastFM Asia graph (7,624 users) with real covariates; treatment and outcome simulated with known effects (direct 2.0, peer 3.0). See make_lastfm_promo.
| effect | method | rel_bias | rmse | coverage | se_over_sd | n_focal |
|---|---|---|---|---|---|---|
| direct | GraphDML (default) | 0.023 | 0.192 | 0.920 | 0.996 | 1613.480 |
| peer | GraphDML (default) | 0.006 | 0.194 | 0.960 | 0.987 | 1613.480 |
| direct | GraphDML (random-order focal set) | 0.019 | 0.222 | 0.930 | 0.948 | 1289.330 |
| peer | GraphDML (random-order focal set) | -0.001 | 0.218 | 0.970 | 1.064 | 1289.330 |
| direct | GraphDML (original procedure) | 0.062 | 0.272 | 0.860 | 0.922 | 1289.330 |
| peer | GraphDML (original procedure) | -0.063 | 0.300 | 0.880 | 1.061 | 1289.330 |
| direct | DML + 1-hop aggregates, all nodes | 0.010 | 0.090 | 0.930 | 0.941 | 7624.000 |
| peer | DML + 1-hop aggregates, all nodes | 0.002 | 0.122 | 0.950 | 1.026 | 7624.000 |
| direct | DML, own covariates (i.i.d.) | 0.557 | 1.118 | 0.000 | 0.995 | 7624.000 |
| direct | OLS + exposure (no network) | 0.371 | 0.749 | 0.000 | 0.938 | nan |
| peer | OLS + exposure (no network) | 0.713 | 2.144 | 0.000 | 0.902 | nan |