How Reliable Is Individualized Functional-Network Attribution in Resting-State fMRI Cognitive Prediction in ADNI?

Introduction

Models that map resting-state functional connectivity (FC) to cognition and clinical phenotypes are increasingly used in Alzheimer’s disease (AD) neuroimaging [1, 2]. Beyond group-level accuracy, a common objective is individualized attribution: for a given subject, estimating which functional networks drive that subject’s prediction. Such per-subject attribution is only meaningful if it is reproducible.

Reliability is a central constraint. Large-scale analyses suggest that stable brain–behavior effects often require much larger samples than historically used [3], and FC test–retest reliability is frequently modest (pooled mean edge-level Intraclass Correlation Coefficient, ICC $\approx0.29$) [4]. The ICC describes how reproducibly a single connectivity edge is measured. Prior work on subject-specific parcellations reports reproducibility across short-interval repeat sessions [5, 6], but does not quantify test–retest reliability of model-derived attribution signatures over the multiyear intervals typical of aging cohorts.

An additional aspect is attribution stability: feature-importance maps can vary substantially under resampling, noise, and other perturbations even when predictive accuracy is similar [7, 8]. In FC prediction, however, attribution work has largely emphasized group-averaged importance. For connectomes rather than attributions, differential identifiability formalizes the person-specific, test–retest-identifiable component [9], and permutation-null models are standard for separating population-shared from person-specific structure [10, 11]. Neither has been applied to per-subject attribution signatures. To the best of our knowledge, no prior work combines per-subject attribution of an FC-based cognitive predictor with a quantitative test–retest evaluation on independent repeat scans of the same individuals.

We address this gap using a frozen FC-based multilayer perceptron (FC-MLP) trained to jointly predict MMSE and ADAS-Cog13 from Schaefer-200 FC on $N=875$ ADNI subjects (Sect. 2). Instead of edge-level ICC, we assess test–retest consistency of each subject’s network-attribution ranking. We compute per-subject network-ablation signatures and evaluate reproducibility with three checks: within-session split-half, scan–rescan across independent repeat visits, and a cluster-aware subject-level permutation null. The main contributions in this study are:

1. Quantify how split-half overestimates scan–rescan reliability of per-subject network attribution (74 subjects; 205 session pairs).

2. Use a subject-level permutation null to estimate the mismatched-subject baseline and the residual person-specific component.

3. Provide reporting recommendations to avoid overstating individual specificity.

Methods

To attribute a prediction to functional networks for a single subject, let $f$ denote the frozen FC-MLP mapping a subject’s connectivity vector $x$ to a two-dimensional prediction $\hat{y}=f(x)$ with targets $t \in\{\mathrm{MMSE},\mathrm{ADAS\text{-}Cog13}\}$. For a given target $t$, we write the scalar prediction as $\hat{y}^{(t)}=[f(x)]_t$.

For each of the seven Yeo networks [12] $n \in\{\mathrm{Vis},\mathrm{SomMot},\mathrm{DorsAttn},\mathrm{SalVentAttn},\mathrm{Limbic},\mathrm{Cont},\mathrm{Default}\}$ (see Figure 1), we ablate the input by zeroing all FC edges incident to nodes in $n$. Because the model input concatenates an FC vector and an edge-coverage mask, $x=[\mathrm{FC};\mathrm{mask}]$, we apply the ablation to both operands: the ablated edges are set to zero in the FC vector and their corresponding mask entries are also set to zero. This yields $x_n$ and $\hat{y}_n^{(t)}=[f(x_n)]_t$. The per-target network importance is then $I_n^{(t)}=\hat{y}^{(t)}-\hat{y}_n^{(t)}$, and the subject-specific importance signature for target $t$ is $\mathbf{I}^{(t)}=(I_1^{(t)},\ldots,I_7^{(t)})$.

This implementation reproduces the group-level ablation results (Figure 1) for all $7 \times2$ network–target combinations to six decimal places. Attribution reliability between two occasions $a, b$ for the same subject is the Spearman rank correlation $\rho(I_a, I_b)$, i.e., the agreement in the within-subject network ranking.

Figure 2 summarizes three reliability checks. (i) Split-half: for each test subject ($n=132$), we split the denoised rs-fMRI time series after scrubbing into odd/even surviving volumes, recompute FC and attribution per half, and compute $\rho(I_{\mathrm{half-A}}, I_{\mathrm{half-B}})$ (minimum 30 volumes per half; 0 exclusions). We additionally test whether split-half $\rho$ increases with per-half scan duration. (ii) Scan–rescan: we attribute independent repeat-visit rs-fMRI and compute $\rho(I_{\mathrm{baseline}}, I_{\mathrm{repeat}})$ for baseline $\times$ all follow-ups, yielding 205 pairs from 74 subjects with uneven repeats ($k_i \in[1, 12]$; 30 subjects contribute 1 pair, 18 contribute 2, 13 contribute 3; tail to 12). (iii) Permutation null: for $B=10{,}000$ permutations, we draw a derangement $\pi$ of the 74 subjects, pair each subject $i$’s baseline signature with the repeat signatures of subject $\pi(i)$ (repeats kept as a block), and recompute the subject-level mean reliability. With $T_{\mathrm{obs}}$ the observed subject-level mean and $b$ the number of permutations with statistic $\ge T_{\mathrm{obs}}$, we report $p=(b+1)/(B+1)$ and $z=(T_{\mathrm{obs}}-\mu_{\mathrm{null}})/\sigma_{\mathrm{null}}$ [11].

Artifact checks. Edge-count: let $\mathbf{c}\in\mathbb{R}^{7}$ be the fixed vector of ablated edge counts (unique edges zeroed when ablating each network). For each subject we compute the across-network Spearman correlation $\rho(\mathbf{I}^{(t)},\mathbf{c})$ (7 items) and summarize it across subjects by its mean, separately per target; with only 7 items, Spearman $\rho$ is highly discrete, so this check is treated as descriptive. For scan–rescan, we first average each subject’s per-pair $\rho(\mathbf{I}^{(t)},\mathbf{c})$ across that subject’s repeat visits and then compute the across-subject mean. Scan duration: we correlate (Spearman, across the 132 test subjects) each subject’s per-half scan duration (seconds; half A, equivalent to half B up to one volume) with split-half $\rho$.

Aggregation and inference. All Spearman correlations are computed separately within each of the three trained runs and averaged across runs. Spearman $\rho$ uses average ranks for ties; no ties occurred in any importance signature (verified for all 1,674 scan–rescan signatures, i.e., 279 sessions $\times$ 2 targets $\times$ 3 runs, and all split-half signatures). As subjects contribute unequal numbers of repeat pairs, scan–rescan is summarized at the subject level by first averaging each subject’s per-pair $\rho$ values and then summarizing across subjects; we also report the pair-pooled mean for comparability. Uncertainty on headline scan–rescan and split-half means is a 95% percentile bootstrap CI over 10,000 resamples of subjects (resampling whole subjects). We test the split-half versus scan–rescan gap with a Wilcoxon signed-rank test across the 74 subjects present in both analyses.

Experimental details

Data description

ADNI is a multi-site, longitudinal study of aging and Alzheimer’s disease with multimodal neuroimaging [13]. We analyzed data from $N=875$ unique ADNI participants who had both T1-weighted MRI and rs-fMRI scans. Within the $N=875$ cohort, diagnoses are CN ($n=464$), MCI ($n=315$), Dementia ($n=94$), and unknown ($n=2$). Age is $72.6 \pm7.9$ years (range 50–97); 489 subjects are female and 386 male. Subjects were enrolled in ADNI1 ($n=37$), ADNI-GO ($n=38$), ADNI2 ($n=256$), ADNI3 ($n=474$), and ADNI4 ($n=68$); 2 unknown. rs-fMRI scans were acquired at 3.0 T across 53 sites with heterogeneous protocols (TR 0.607–6.02 s; 96–972 volumes after removal of initial volumes). For the upstream prediction model, the cohort is split once at the subject level into fixed train/validation/test partitions of 611/132/132 stratified by diagnosis. We evaluate attribution and reliability only on the held-out 132-subject test split; scan–rescan further restricts to test subjects with repeat-visit rs-fMRI.

Implementation details

We analyze a trained FC-MLP and keep its weights fixed for all attribution and reliability analyses.

Input features. rs-fMRI preprocessing includes motion correction, HD-BET brain extraction [14], rigid rs-fMRI→T1 registration, detrending, and nuisance regression of mean white-matter (WM) and cerebrospinal-fluid (CSF) signals plus 5 anatomical CompCor (aCompCor) components [15] from a combined WM/CSF mask, without global signal regression. Time series are band-pass filtered (0.01–0.1 Hz, using each:VEVENT subject’s TR, $z$-scored, and scrubbed at framewise displacement (FD) > 0.5 mm after filtering; FD was computed from translation magnitude only rather than the full six-parameter formulation [16], as rotation parameters were not retained upstream. No spatial smoothing or slice-timing correction is applied. We compute Schaefer-200 [17] FC as a Fisher-$z$ transformed $200 \times200$ Pearson correlation matrix and vectorize the upper triangle (19,900 edges). An edge-coverage mask is concatenated, yielding a 39,800-dimensional input.

Model and training (frozen). The FC-MLP encoder maps $39{,}800 \to256 \to128$ with LayerNorm/GELU/dropout (0.3), and the decoder maps $128 \to64 \to2$ with GELU/dropout (0.3). The model is trained with Huber loss ($\delta=1.0$) using Adam (lr = $10^{-4}$, weight decay $10^{-5}$; no schedule/warmup), batch size 16, and up to 100 epochs. Early stopping uses patience 15 on validation loss; we keep the single best checkpoint (lowest validation loss). During training only, an augmentation zeros 5 randomly chosen parcels per sample. Targets are $z$-scored on train only and inverse-transformed for reporting. Three independent runs with different random initializations share the same fixed split. ComBat harmonization [18] uses the TR group as batch variable (a proxy for acquisition protocol) and protects diagnosis, whose composition differs across TR groups ($\chi^2=35.66$, $p < 0.001$); no other covariates are included. It is fit on the training split only, never refit, and applied to validation/test subjects and split-half halves, but not to repeat-visit sessions (both sides of scan–rescan use pre-ComBat features).

Results

The frozen predictor is weak: on the $n=113$ test subjects with both targets, $R^2=0.053 \pm0.034$ for MMSE and $0.087 \pm0.037$ for ADAS-Cog13 (mean $\pm$ SD over runs). As a diagnostic check, the predicted ADAS-Cog13 score separates CN from Dementia with AUC 0.780 (95% bootstrap CI 0.611–0.911; $n=72$ of the 113 subjects with both targets: 58 CN, 14 Dementia).

Within-session split-half reliability is near-ceiling for both targets (mean $\rho=0.948/0.950$, median $\approx0.99$; Table 1). On independent repeat visits (205 session pairs from 74 subjects), subject-level scan–rescan reliability drops to $\rho=0.609$ (MMSE) and 0.616 (ADAS-Cog13), with large between-subject variation (10th–90th percentiles 0.34–0.90 and 0.33–0.85). Pair-pooled means are slightly lower (0.593/0.588) because subjects with more follow-ups receive more weight. Restricted to the same 74 subjects, split-half reliability is 0.933/0.939, a paired difference of 0.324/0.323 (Wilcoxon $p < 10^{-12}$), indicating that split-half substantially overestimates test–retest reliability.

Statistic	Split-half MMSE	Split-half ADAS-Cog13	Scan–rescan MMSE	Scan–rescan ADAS-Cog13
Mean	0.948	0.950	0.609	0.616
95 % CI	0.929–0.964	0.934–0.964	0.553–0.663	0.571–0.660
Median	0.988	0.988	0.625	0.661
Std	0.101	0.085	0.242	0.198
10–90%	0.87–1.00	0.85–1.00	0.34–0.90	0.33–0.85
Run SD ($n=3$)	0.005	0.007	0.033	0.020
Null mean ± SD	–	–	0.463 ± 0.021	0.472 ± 0.022
Null range	–	–	0.379–0.549	0.393–0.562
$z$-score	–	–	6.89	6.63
$p$-value	–	–	≤ $10^{-4}$	≤ $10^{-4}$
Pair-pooled mean	–	–	0.593	0.588

Under the subject-level permutation null, mismatched subjects still correlate strongly ($\rho=0.463/0.472$), but the matched statistic exceeds the null ($z=6.89/6.63$; $p \le10^{-4}$), corresponding to a modest person-specific excess of $\approx0.15$ (MMSE) and $\approx0.14$ (ADAS-Cog13). The artifact checks do not indicate a network-size effect: mean edge-count correlations $\rho(\mathbf{1}^{(t)}, c)$ are weakly negative for split-half (MMSE −0.120; ADAS-Cog13 −0.163) and repeat visits (MMSE −0.112; ADAS-Cog13 −0.158). Split-half reliability increases with per-half scan duration ($\rho=0.199/0.277$), consistent with sampling-noise limits, although duration is confounded with acquisition protocol in ADNI.

Discussion

Per-subject attribution reliability depends strongly on protocol: split-half is near-ceiling ($\rho\approx0.95$), but drops on scan–rescan across independent repeat visits ($\rho\approx0.61$–$0.62$), consistent with additional between-visit variability beyond within-session sampling noise. The permutation null suggests that much of this structure is population-shared, with only a modest person-specific excess; robustness to other cohorts, preprocessing, parcellations, attribution operators, and models remains untested.

The coexistence of shared and person-specific structure mirrors FC itself, where connectomes are identifiable across sessions [9, 10] yet test–retest reliability is modest [4]. The strong shared profile likely has two sources. First, the ablations overlap by construction: every between-network edge $(i,j)$ with $i \in n$, $j \in m$ is zeroed in both $x_n$ and $x_m$, coupling $I_n^{(t)}$ and $I_m^{(t)}$; within-network-only ablation would remove this overlap. Second, the weak predictor may respond mainly to group-level FC structure. Because $\rho$ is bounded above, a high shared baseline leaves little room for an individual excess, so the person-specific component may be underestimated.

Limitations include the modest repeat-visit sample (74 subjects; 205 session pairs) and the weak, run-variable predictor (Sect. 3). Repeat-visit sessions were not ComBat-harmonized, so part of the split-half to scan–rescan gap may reflect this mismatch. Inter-visit intervals are long and variable (median 24.4 months, IQR 12.2–48.4) and are not modeled.

We therefore recommend treating within-session proxies as expected upper bounds, reporting scan–rescan estimates when repeat visits exist, and including a permutation-null (or equivalent) control; these recommendations should be validated in further aging/AD cohorts such as OASIS-3 and AIBL. In summary, per-subject network attributions are far less reproducible across visits than within-session estimates suggest, and most of their apparent stability reflects a pattern shared across the cohort, so individual specificity should be demonstrated against an explicit population baseline before attributions are interpreted as personal biomarkers.

Acknowledgement. We acknowledge HPC support from XXXXXX and partial hardware funding from the XXXXX. Data were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI; adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design of ADNI but did not participate in this report. Code, derived features, and the subject-level split file are available upon reasonable request, subject to ADNI DUA constraints.