betaFeedback

WorksEngineering

CoupleFlow: Null-Calibrated Auditory Attention Decoding With Time-Locked Brain-Speech Coupling and Flow-Matching Likelihood Ratios

Abstract

Auditory attention decoding (AAD) seeks to identify the attended talker from brief noninvasive brain recordings by comparing neural responses with competing speech streams. Recent electroencephalography (EEG) decoders report nearly perfect accuracy from one second of data, yet we show that this performance fails to generalize to unseen speech because models exploit trial leakage, eye gaze, and acoustic cues instead of speech evoked neural activity. We introduce CoupleFlow, which decodes attention through temporal coupling between neural and acoustic embeddings. Its predictions remain at chance when the brain signal contains no informative coupling, preventing shortcuts based on speech content or talker identity. A conditional flow matching head implements a likelihood ratio decision rule based on pointwise mutual information between brain and speech. This rule is Bayes optimal when the recording depends only on the attended talker and removes preferences for speech that is likely a priori. We evaluate CoupleFlow with content disjoint splits recovered from the audio and null tests that distinguish genuine temporal coupling from trial recognition. Across nine EEG and magnetoencephalography (MEG) datasets processed through a common pipeline, spanning four languages, high density EEG recordings, listeners with hearing impairment, and a room with multiple talkers, CoupleFlow outperforms linear and deep baselines while all null tests remain at chance. Finally, we derive an evidence accumulation law that predicts accuracy across window lengths from a single parameter per dataset and predicts that parameter from the strength of neural speech tracking. This law identifies recording signal to noise ratio as a limit on decoding speed and reliability and estimates the window length required to reach a target accuracy.

Introduction

When several people talk at once, a listener naturally isolates one voice. The auditory cortex reflects this selection: low-frequency cortical activity tracks the attended talker’s speech envelope far more strongly than ignored speech. [15, 32, 37]. Auditory attention decoding (AAD) asks whether the attended talker can be identified from this auditory evoked non-invasively, with electroencephalography (EEG) or magnetoencephalography (MEG). EEG is promising because it offers millisecond time scale resolution and works with a few electrodes near the ear [34, 17]. This makes AAD a candidate for neuro-steered hearing devices that amplify the attended talker [22], since hearing-impaired listeners track the attended talker more strongly than the competing one, as normal-hearing listeners do [38, 13, 19]. Building such a device is beyond the scope of this paper, but we study the decoding problem in terms of its ability to generalize to unheard speech using limited data from unseen subjects, solely from brain activity.

Using a few seconds of multichannel neural data and the speech of KK candidate talkers, the decoder identifies the attended voice. The problem is difficult because the neural response to speech is weak (a linear decoder’s reconstruction correlates with the attended envelope at only r=0.07r = 0.07–0.170.17 on our two-talker EEG datasets), differs across listeners, and is mixed with activity unrelated to the audio stimulus, while the candidates often differ in ways a decoder can exploit without reading the brain: loudness, voice, or the identity of the recording session.

AAD decoders follow two strategies: Stimulus reconstruction and Direct classification. Stimulus reconstruction uses the EEG to estimate of the attended speech envelope, and it selects the candidate talker that correlates best with this envelope. Direct classification maps EEG to a label, usually the side of the attended talker. In both cases, however, performance usually fails short of expectations (details in App. § 7). With stimulus reconstruction, linear backward models and canonical correlation analysis [37, 10, 12] are robust, but they require long signal durations (i.e., tens of seconds) for high accuracy, which is not practical. Deep nonlinear reconstructions have not changed this picture. The best of them still reconstruct the envelope weakly (r≈0.19r \approx0.19 for VLAAI; [3]), nonlinear decoders did not consistently outperform linear ones in an independent comparison on two public datasets [22], and many reported gains come from evaluations in which training and test data overlap [39]. Conversely, direct classifiers [41, 44, 16] (Ni et al. 2024) report 95–97% at 1 s, but their accuracy rests largely on shortcuts: training and test windows come from the same trial, so the classifier can recognise the trial rather than decode attention (leakage), and it can read eye gaze toward the attended talker. When training and test data come from different trials, their accuracy drops to near chance [40]. Furthermore, datasets replay the same stories to every listener, and test windows are often cut from training trials [39]. It is also not inherently clear that decoders are not overfitting to audio cues, such as loudness, spectral differences, or trial identity, indicating that the decoder is ignoring the brain signal. Most studies report accuracy at only one or two window lengths, so the window needed for a given accuracy cannot be predicted. Each study also preprocesses each dataset its own way, so numbers cannot be compared across studies or datasets. Finally, in one widely used dataset (NJU) we find that the EEG is not time-aligned with the audio at any offset, so results on it cannot reflect speech tracking (App. § 9).

Our approach addresses the AAD problem by ensuring that decoding is driven only by the brain’s time-locked response to speech, while quantifying the effects of window length and recording quality on accuracy. Our approach scores candidates based solely on the temporal covariance between learned acoustic and EEG embeddings. The EEG representation is centered so that signals without time-varying information assign equal scores to all candidates. A flow-matching model adds a likelihood-ratio decision rule, while content-disjoint splits and null analyses quantify the contribution of neural activity to decoding performance.

Contributions. (i) A decoder that can only use the brain’s response to speech. CoupleFlow (§ 2) chooses the attended talker using only how the EEG follows each talker’s speech over time. It is therefore guaranteed to be at chance when the EEG carries no information about the speech.

information about the speech. Examples are EEG from a different trial, EEG shifted in time so that it no longer lines up with the speech, or random noise. Differences in loudness, voice or trial identity cannot raise its accuracy. A generative head adds a new decision rule for AAD, which is optimal under a standard assumption.

(ii) A leakage-free benchmark. We evaluate on nine public EEG and MEG datasets under one preprocessing pipeline. Test speech is never heard in training, and control tests check that accuracy comes from the brain. Under this protocol, CoupleFlow outperforms linear and deep baselines trained on the same data, and we show where previously reported accuracies came from.

(iii) A law for accuracy versus window length. A simple formula predicts decoding accuracy at any window length from one number per dataset, and predicts that number from how strongly the brain tracks speech. It shows that fast, reliable decoding is limited by the quality of the recording, and it gives the window needed for any target accuracy.

CoupleFlow

A decision window of TT seconds (L=64TL = 64T samples at 64 Hz) gives the recording x∈RC×Lx \in\mathbb{R}^{C \times L} of CC channels, the speech envelopes a1,…,aK∈RLa_1,\ldots,a_K \in\mathbb{R}^{L} of KK candidate talkers, and the index yy of the attended one. Figure 1 shows how CoupleFlow turns them into a decision. A neural path (blue) embeds the recording and an acoustic path (orange) embeds each candidate, both as time series of the same size. The two paths meet only in a coupling score (green), which makes the decision, and a flow head (pink) supplies a second decision rule. App. § 10 gives every layer.

CoupleFlow architecture

Figure 1. CoupleFlow. The two small signal plots show the input of one real MAESTRO test window (listener S03): seven of its EEG channels and the envelopes of its K=4K = 4 candidate talkers, with the attended talker in dark orange. The neural path (blue) maps the EEG xx to an embedding ee whose value at time tt sees only EEG from t+δt + \delta to t+δ+0.98 st + \delta+ 0.98\,\mathrm{s} after the sound (δ∈{0,125} ms\delta\in\{0,125\}\,\mathrm{ms}). The acoustic path (orange) embeds each candidate kk with one shared encoder. The paths meet only in the coupling score sks_k (Eq. (1)), which makes the decision. The flow head (pink) supplies the alternative pointwise-mutual-information (PMI) rule (Prop. 2) and per-source residuals. Tags mark where each loss in Eq. (2) acts, coloured by role: decoding (green), density (pink), anti-shortcut (gold), auxiliary (grey). Dashed: reversed gradient of the audio-only adversary.

The EEG first goes through a listener correction, a small low-rank adjustment added to a model shared by all listeners. It absorbs individual differences such as head shape and cap placement without fitting a separate model to each listener’s few minutes of data. A layer of virtual sensors then reduces the channels to a few learned spatial filters, as backward models do, and a lag-window encoder turns them into the neural embedding ee. Each time step of ee sees only the second of EEG that follows the corresponding sound (from a lag δ∈{0,125}\delta\in\{0,125\} ms on), which is where a cortical response to that sound must lie. EEG before the sound cannot be a response to it, and serves as our lag control.

The acoustic path starts from candidate features: each talker’s envelope and onsets and, where the audio is available, sub-band envelopes and glimpsed/masked channels that mark where the talker is louder or quieter than the competing one [8]. In training, a quantile match gives the candidates’ envelopes identical value distributions, which removes loudness and level cues. A shared encoder, with same weights for every candidate, then maps each candidate to an embedding a^k\hat{a}_k of the same size as ee. It sees only past audio, so the brain response it is compared with must follow the sound.

The coupling score measures how each candidate’s embedding co-varies in time with the neural embedding:

s(e,a^)=1τ∑d=116ωd⟨e⋅d−eˉd,a^⋅d−a^ˉd⟩∥e⋅d−eˉd∥∥a^⋅d−a^ˉd∥,(1)s(e,\hat{a}) = \frac{1}{\tau}\sum_{d=1}^{16}\omega_d\frac{\langle e_{\cdot d}-\bar{e}_d,\hat{a}_{\cdot d}-\bar{\hat{a}}_d\rangle}{\lVert e_{\cdot d}-\bar{e}_d\rVert\lVert\hat{a}_{\cdot d}-\bar{\hat{a}}_d\rVert}, \tag*{(1)}

where e⋅d∈RLe_{\cdot d} \in\mathbb{R}^{L} is dimension dd of ee over time and eˉd\bar{e}_d its time average (likewise for a^\hat{a}). Each fraction is the Pearson correlation over time between one neural and one acoustic dimension; the weights ωd\omega_d and the temperature τ\tau are learned. Candidate kk scores

sk=s(e,a^k)−1K∑js(e,a^j),s_k = s(e,\hat{a}_k) - \frac{1}{K}\sum_j s(e,\hat{a}_j),

and the decision is y^=arg⁡max⁡ksk\hat{y} = \arg\max_k s_k. Because the correlation removes each dimension’s time average, a neural embedding that does not vary in time gives every candidate the score 0. The audio alone then cannot favour any candidate, and the decoder is at chance whenever the EEG carries no information about the speech (Prop. 1).

Alongside the coupling score, the flow head, a conditional flow-matching network [31], learns to generate the attended envelope from ee. Its condition is dropped in part of the training windows (Ho and Salimans, 2021), so the same network also learns to generate envelopes without the brain. At test time each candidate passes through it with and without ee, and how much the recording lowers the network’s error approximates the pointwise mutual information

PMI⁡(ak;x)=log⁡p(ak∣x)−log⁡p(ak).\operatorname{PMI}(a_k;x)=\log p(a_k\mid x)-\log p(a_k).

Picking the candidate with the largest PMI is a second decision rule. It is Bayes-optimal when the recording depends only on the attended talker (Prop. 2), and since log⁡p(ak)\log p(a_k) cancels, no candidate can win by being typical speech.

The model is trained end to end on

L=LCE+λNCELNCE⏟decoding+λFMLFM⏟density+λnullLnull+λadvLadv+λVICLVIC⏟anti-shortcut+λsideLside+λenvLenv⏟auxiliary,(2)\mathcal{L}=\underbrace{\mathcal{L}_{\mathrm{CE}}+\lambda_{\mathrm{NCE}}\mathcal{L}_{\mathrm{NCE}}}_{\text{decoding}}+\underbrace{\lambda_{\mathrm{FM}}\mathcal{L}_{\mathrm{FM}}}_{\text{density}}+\underbrace{\lambda_{\mathrm{null}}\mathcal{L}_{\mathrm{null}}+\lambda_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}}+\lambda_{\mathrm{VIC}}\mathcal{L}_{\mathrm{VIC}}}_{\text{anti-shortcut}}+\underbrace{\lambda_{\mathrm{side}}\mathcal{L}_{\mathrm{side}}+\lambda_{\mathrm{env}}\mathcal{L}_{\mathrm{env}}}_{\text{auxiliary}}, \tag*{(2)}

and the tags in Figure 1 mark where each term acts. The decoding terms ask for the right decision, through cross-entropy over the candidates and a contrastive term that matches each window’s EEG to its own attended speech among all windows of the batch. The anti-shortcut terms close routes around the brain: a hinge requires the attended speech to score higher with its own recording than with another window’s, an audio-only adversary, through a reversed gradient, strips acoustic cues to the attended talker from the shared encoder, and a VICReg term keeps the neural embedding from collapsing. The density term trains the flow head. The auxiliary terms train a side head, which predicts the attended side from the EEG in training only, and an envelope read-out that reports tracking (Table 2); neither enters a reported decision. Checkpoints are selected on validation contribution, accuracy minus the across-trial null, which rewards what the brain adds rather than what the audio or trial identity allows. App. § 11 defines every term, gives its weight (set once for all datasets) and reports the effect of removing it.

DatasetLinear (trials) rattr_{\mathrm{att}}Linear (trials) rcompr_{\mathrm{comp}}Ours (5 s) rattr_{\mathrm{att}}Ours (5 s) rcompr_{\mathrm{comp}}
KU Leuven0.0990.0270.1330.024
DTU0.1300.0230.1420.025
Fuglsang’200.1710.0180.1480.006
ESAA0.1390.0190.1360.021
KUL-255ch0.1010.0180.1150.024
Broderick0.0730.0030.0820.048
SparrKULee0.132–0.172–
MEG-MASC0.278–0.396–

Table 2. Envelope tracking rr with the attended and the competing envelope. Linear: per-listener backward model, held-out trials (App. Table 20). Ours: linear read-out of the neural embedding (LenvL_{\mathrm{env}}), 5-s test windows, content-disjoint folds, mean of 3 seeds.

Theory

Setting. A decision window gives a recording xx, candidate envelopes a1:K=(a1,…,aK)a_{1:K} = (a_1,\ldots,a_K) and the attended index y∈{1,…,K}y \in\{1,\ldots,K\}, drawn jointly. A decoder gives candidate kk a score sks_k and predicts y^=arg maxksk\hat{y} = \mathop{\mathrm{arg\,max}}_k s_k; a tie for the maximum counts as an error.

Proposition 1 (null calibration). Assume (a) sk=g(x,ak)s_k = g(x,a_k) for one function gg shared by all candidates; (b) H0H_0: xx is independent of (y,a1:K)(y,a_{1:K}); (c) given yy, the joint law of a1:Ka_{1:K} is invariant to permuting the candidate indices. Then under H0H_0, P(y^=y)≤1/K\mathop{\mathrm{P}}(\hat{y}=y) \le1/K.

Eq. (1) satisfies (a); the across-trial and circular nulls realise (b).

Proposition 2 (PMI is Bayes-optimal). Let p(a)p(a) be the density of the attended envelope aya_y and p(a∣x)p(a \mid x) its density given the recording, and PMI⁡(a;x)=log⁡p(a∣x)−log⁡p(a)\operatorname{PMI}(a;x) = \log p(a \mid x) - \log p(a). Assume (a) the recording depends on the candidates only through the attended one, p(x∣a1:K,y)=p(x∣ay)p(x \mid a_{1:K},y) = p(x \mid a_y); (b) yy is uniform on {1,…,K}\{1,\ldots,K\} and independent of a1:Ka_{1:K}. Then the maximum a posteriori (MAP) decision is y^=arg maxkPMI⁡(ak;x)\hat{y} = \mathop{\mathrm{arg\,max}}_k \operatorname{PMI}(a_k;x). Scoring by log⁡p(ak∣x)\log p(a_k \mid x) instead adds log⁡p(ak)\log p(a_k), a term that depends on the audio alone.

Proposition 3 (evidence accumulation). Let a decoder output a time series a~\tilde{a} over a window of TT s (e.g. a reconstructed envelope), let ratt,rnegr_{\mathrm{att}},r_{\mathrm{neg}} be its sample Pearson correlations with the attended and the other (competing, or match–mismatch) envelope and ρatt,ρneg\rho_{\mathrm{att}},\rho_{\mathrm{neg}} their population values, and let the decision be correct when ratt>rnegr_{\mathrm{att}} > r_{\mathrm{neg}}. Assume (i) the window holds neff=fTn_{\mathrm{eff}} = fT effectively independent samples (ff per second); (ii) the Fisher transforms zj=atanh⁡rj∼N(ζj,1neff−3)z_j = \operatorname{atanh} r_j \sim\mathcal{N}\left(\zeta_j,\frac{1}{n_{\mathrm{eff}}-3}\right) with ζj=atanh⁡ρj\zeta_j = \operatorname{atanh}\rho_j, j∈{att,neg}j \in\{\mathrm{att},\mathrm{neg}\}; (iii) zatt,znegz_{\mathrm{att}},z_{\mathrm{neg}} are independent. With Δζ=ζatt−ζneg\Delta\zeta= \zeta_{\mathrm{att}}-\zeta_{\mathrm{neg}},

P(correct∣T)=Φ(ΔζfT−32)≈Φ(κT),P(\mathrm{correct}\mid T)=\Phi\left(\Delta\zeta\sqrt{\frac{fT-3}{2}}\right)\approx\Phi(\kappa\sqrt{T}),

with κ=Δζf/2\kappa=\Delta\zeta\sqrt{f/2}, and accuracy pp needs Tp=(Φ−1(p)/κ)2T_p=(\Phi^{-1}(p)/\kappa)^2 s. If further a~\tilde{a} is a signal plus independent noise with per-sample signal-to-noise ratio (SNR) σ\sigma (variance ratio), and ρj∗\rho_j^* are the noise-free correlations, then ρj=ρj∗σ/(1+σ)\rho_j=\rho_j^*\sqrt{\sigma/(1+\sigma)} and, for small ρj\rho_j, κ2∝σ/(1+σ)\kappa^2\propto\sigma/(1+\sigma).

Proofs are in App. § 8. Prop. 3 makes κ\kappa a single, window-independent figure of merit; for CoupleFlow, whose score is not a single correlation, we fit κ\kappa per curve. Model quality and training data act through the noiseless correlation, and recording quality through σ\sigma.

Experiments

Data. We test on nine public datasets (Table 7), chosen so that a gain cannot come from one recording setup. Six are two-talker EEG studies in Danish, Dutch, Mandarin and English: KU Leuven (Das et al., [11]), DTU (Fuglsang et al., [18]), Fuglsang’20 (Fuglsang et al., [19]), whose 44 older listeners include 22 with hearing loss, ESAA (Li et al., 2022), and two high-density sets with 255 (Mundanad Narayanan et al., [35]) and 128 electrodes (Broderick et al., [9]). SparKULee (Accou et al., [1]) and MEG-MASC (Gwilliam et al., [24]) have a single talker. They test whether a decoder can tell the heard speech from other speech, and MEG-MASC shows how far decoding goes when the recording is much cleaner than EEG. MAESTRO (Hassan et al., [25]) adds a room in which the listener attends one of four talkers.

Datasetlistenerssensorslanguagetaskhoursgroupsfoldsenv.
KU Leuven ([11, 6])1664 EEGDutch2-talker20.644ours
DTU ([18])1864 EEGDanish2-talker15.0605ours
Fuglsang’20 ([19])44 (22 HI)64 EEGDanish2-talker20.144prov.
ESAA ((Li et al., 2022))1764 EEGMandarin2-talker10.9325ours
KUL-25ss ((Mundanarayanan et al., 2021))30253 EEGDutch2-talker13.222ours
Broderick ([9])33128 EEGEnglish2-talker16.4305prov.
SparrKULee ([1])8564 EEGDutch1-talker132.4635ours
MEG-MASC ([24])27208 MEGEnglish1-talker51.4305ours
MAESTRO ([25])1632 EEGEnglish4 of 6 talkers13.11005 / LOSOours

Table 7. Datasets after unified preprocessing.

Preprocessing. Published results preprocess each dataset differently, so differences between datasets partly reflect the pipeline. We therefore run every raw recording through one pipeline (App. § 9). It keeps 0.5–32 Hz, which contains the slow cortical tracking of the speech envelope (Ding and Simon, [15]), and removes drift and most muscle noise. It maps every cap onto the same 64 channels, so one network serves all datasets, and resamples to 64 Hz, enough for this band. We compute the speech envelope with a gammatone filterbank, a standard model of the cochlea; two datasets ship no audio, and for them we use the authors’ envelopes. Finally, we check that the EEG of every dataset tracks its audio at a plausible latency. One widely used dataset fails this check and is excluded (App. § 9).

Protocol. Test speech is never heard in training. Most datasets play the same stories to every listener, and some replay them, so holding out trials does not hold out speech. We therefore group the trials that contain the same speech, found by correlating their envelopes since the releases lack a reliable stimulus key, and build each test fold from whole groups (content-disjoint folds; App. § 9). We report three candidate sets: competing (comp.), the attended vs. the other talker; matched, the same pair after quantile matching, which removes loudness cues; and same-talker (match–mismatch, mm), the attended speech vs. the same talker’s speech from another window of the trial, the only task for single-talker data. Every accuracy comes with nulls that make the EEG uninformative: another trial’s recording of the same listener (across-trial) and, at 5 s, the trial’s own recording shifted by at least 2 s (circular\textit{circular}), which keeps trial identity but breaks time-locking. Baselines. We compare with one decoder from each main family, all trained on the same folds, windows and inputs and stopped early on validation data: the per-listener linear backward model, still the standard for AAD; VLAAI [3], a deep reconstruction model trained across listeners; and a dilated-convolution match–mismatch model (Dil. MM; [2]), which, like CoupleFlow, compares the EEG with each candidate directly. A recursive decoder (DECAF) stays near chance on the two-talker sets and appears only in App. Table 13; training details are in App. C.

Datasetmodel1 s2 s5 s10 s20 s30 s60 strial
KU LeuvenLinear57.060.064.969.975.179.284.287.8
KU LeuvenVLAAI56.959.363.868.573.077.482.688.1
KU LeuvenDilated MM55.359.865.971.678.282.689.896.9
KU LeuvenDECAF50.951.651.352.055.454.158.365.0
KU LeuvenDECAF (EEG branch)51.351.449.150.251.051.250.355.3
KU LeuvenCoupleFlow60.565.072.779.386.790.596.499.3
KU LeuvenCoupleFlow, mm64.870.679.787.393.896.699.0–
DTULinear59.163.972.479.086.190.0–93.9
DTUVLAAI57.560.464.869.774.178.2–82.2
DTUDilated MM56.061.469.175.383.788.5–92.9
DTUDECAF50.050.050.350.150.148.8–49.6
DTUDECAF (EEG branch)51.051.149.750.051.051.2–52.5
DTUCoupleFlow61.467.476.283.490.694.5–97.0
DTUCoupleFlow, mm64.171.081.989.1––––
Fuglsang’20Linear63.468.577.385.191.994.6–98.3
Fuglsang’20VLAAI60.663.768.974.581.785.4–89.8
Fuglsang’20Dilated MM57.964.373.080.289.092.7–95.4
Fuglsang’20DECAF48.747.941.437.032.519.8–39.7
Fuglsang’20DECAF (EEG branch)51.251.346.645.343.841.7–39.5
Fuglsang’20CoupleFlow63.869.879.686.793.395.9–98.5
Fuglsang’20CoupleFlow, mm66.373.183.090.295.7–––
ESAALinear61.365.674.180.387.690.293.794.7
ESAAVLAAI58.361.566.471.878.881.788.991.0
ESAADilated MM57.562.770.677.384.787.392.794.3
ESAADECAF50.049.949.950.050.050.550.349.8
ESAADECAF (EEG branch)51.151.549.749.449.748.952.250.7
ESAACoupleFlow63.268.977.984.990.693.496.197.1
ESAACoupleFlow, mm62.668.777.184.691.292.8––
KUL-255Linear57.060.565.570.976.281.587.498.3
KUL-255VLAAI55.958.062.167.072.175.482.295.0
KUL-255Dilated MM54.758.865.070.477.682.588.598.3
KUL-255DECAF50.150.150.450.949.850.151.252.5
KUL-255DECAF (EEG branch)51.050.949.950.951.153.152.252.5
KUL-255CoupleFlow58.962.869.475.983.087.793.999.4
KUL-255CoupleFlow, mm61.165.873.580.587.691.996.0–
BroderickLinear56.559.163.768.875.978.8–84.3
BroderickVLAAI51.051.352.253.153.153.9–55.4
BroderickDilated MM50.551.352.352.754.453.6–56.2
BroderickDECAF50.150.050.350.250.850.5–50.6
BroderickDECAF (EEG branch)50.650.650.350.052.250.8–49.2
BroderickCoupleFlow55.859.063.669.075.279.2–83.1
BroderickCoupleFlow, mm55.157.862.365.670.2–––
SparrKULeeLinear61.867.276.383.790.593.597.4–
SparrKULeeVLAAI59.964.070.577.183.987.593.3–
SparrKULeeDilated MM62.671.583.391.096.498.399.7–
SparrKULeeDECAF53.154.656.057.860.963.066.7–
SparrKULeeDECAF (EEG branch)54.556.559.261.866.669.073.2–
SparrKULeeCoupleFlow73.582.392.297.199.399.7100.0–
MEG-MASCLinear72.781.392.097.399.499.8100.0–
MEG-MASCVLAAI71.078.789.195.698.699.6100.0–
MEG-MASCDilated MM72.085.096.299.3100.0100.0100.0–
MEG-MASCDECAF62.969.980.689.095.897.499.7–
MEG-MASCDECAF (EEG branch)63.269.980.689.695.997.899.7–
MEG-MASCCoupleFlow83.793.399.099.9100.0100.0100.0–

Table 13. CoupleFlow and every baseline at every window (%, content-disjoint folds, pooled over all test windows; competing talker for two-talker sets, match–mismatch for SparrKULee and MEG-MASC). CoupleFlow: mean of 3 seeds with the validation-chosen lag window, as in Table 1; “CoupleFlow, mm”: match–mismatch on the two-talker recordings. trial: one decision per trial. “–”: window longer than the trials. CoupleFlow nulls: 49.7–50.5% up to 10 s; 42.5–61.1% at 20–60 s, where few windows exist. DECAF, trained with the same folds and selection rule, stays near chance on the two-talker sets.

Main results (Table 1; leave-one-listener-out, LOSO: listener and content held out together). At 5 s CoupleFlow is best on seven of eight datasets. Its margin over the strongest baseline is +6.8 points on KU Leuven, +3.8 on DTU, +2.3 on Fuglsang’20, +3.8 on ESAA, +3.9 on the 255-channel set, +8.9 on SparrKULee and +2.8 on MEG. On Broderick it ties the per-listener linear decoder (-0.1, within one seed s.d.) and stays within one point of it at every window. On the 5 other two-talker EEG sets the margin grows from 1 to 20 s, to at most +8.5 points. On the single-talker match–mismatch sets the margin is largest at 1 s (at least 10.9 points) and shrinks as both models approach 100%. On unseen listeners CoupleFlow keeps 2.0–8.3 points over the pooled linear model wherever either transfers; on Broderick neither does (App. F.6). The input features were chosen during development on KU Leuven fold 0 (App. C); without that fold KU Leuven scores 72.8% at 5 s. Lag window. The lag δ\delta is the one hyperparameter set per dataset, on validation accuracy only (App. C): 0 ms for DTU, ESAA, the 255-channel set, SparrKULee, MEG and MAESTRO, 125 ms otherwise. It matters most on ESAA, whose audio onset had to be estimated (71.2% at 5 s with 125 ms; App. Table 19).

Datasettask1 s LinearVLAAIDil.MMOurs5 s LinearVLAAIDil.MMOurs20 s LinearVLAAIDil.MMOursLOSO, 5 s LinearOurs
KU Leuvencomp.57.056.955.360.564.963.865.972.775.173.078.286.759.768.0
DTUcomp.59.157.556.061.472.464.869.176.286.174.183.790.664.070.4
Fuglsang’20comp.63.460.657.963.877.368.973.079.691.981.789.093.367.573.7
ESAAcomp.61.358.357.563.274.166.470.677.987.678.884.790.666.168.6
KUL-255chcomp.57.055.954.758.965.562.165.069.476.272.177.683.064.066.0
Broderickcomp.56.551.050.555.863.752.252.363.675.953.154.475.249.250.1
SparrKULeemm61.859.962.673.576.370.583.392.290.583.996.499.3–85.9
MEG-MASCmm72.771.072.083.792.089.196.299.099.498.6100.0100.0–95.0

Table 1. Accuracy (%) of CoupleFlow (Ours) and the baselines at 1, 5 and 20 s on the content-disjoint folds (every model trained and tested on the same folds), and on unseen listeners (LOSO: listener and content held out, 5 s) against the pooled linear backward model. comp.: attended vs. competing talker; mm: match–mismatch. Ours: mean of 3 seeds (s.d. 0.0–1.8 at 5 s); LOSO: one seed. Bold: best per group. CoupleFlow’s EEG-shuffle nulls lie in [49.7, 50.5] up to 10 s. All windows and per-trial accuracy: App. Table 13; full LOSO results: App. Table 21.

The accumulation law holds. Figure 2: one κ\kappa per curve fits all 14 dataset–task curves from 1 to 60 s with mean absolute error 0.2–1.8 points, and 0.4–2.0 points when predicting a held-out window. EEG gives κ=0.13\kappa= 0.13–0.630.63 (T95T_{95}, the window for 95% accuracy, = 7–170 s). MEG gives 1.02 (T95≈2.6T_{95} \approx2.6 s). All native channels raise κ\kappa from 0.140 to 0.181 (Broderick) and from 0.190 to 0.224 (253 channels; App. Table 10).

Figure 2. Accuracy vs. window length (log scale). Markers: CoupleFlow (seed 0), pooled over folds. Lines: Φ(κT)\Phi(\kappa\sqrt{T}) with one κ\kappa per curve (Prop. 3).

Why does MEG reach κ≈1\kappa\approx1: the sensor, or the easier task? Prop. 3 separates the two. A match–mismatch negative is uncorrelated with the response (ρneg=0\rho_{\mathrm{neg}} = 0), while a competing talker is weakly tracked, ρneg=γρatt\rho_{\mathrm{neg}} = \gamma\rho_{\mathrm{att}} with γ\gamma the ignored-talker ratio, so for small ρ\rho the task alone predicts κcomp/κmm=1−γ\kappa_{\mathrm{comp}}/\kappa_{\mathrm{mm}} = 1 - \gamma. With γ\gamma measured by the per-listener linear decoder (Table 2), this holds on all six two-talker datasets (mean error 0.02, r=0.95r = 0.95): the task raises κ\kappa by a factor 1.05–1.38, not more. With one effective rate f=9.7f = 9.7 samples/s, κ=f/2Δζ\kappa= \sqrt{f/2\Delta\zeta} fits all 14 dataset–task values of the linear decoder (r=0.996r = 0.996; Figure 4), and MEG lies on the same line as EEG. Its advantage is therefore a per-sample signal-to-noise effect (details in App. F.4). CoupleFlow’s envelope read-out, an auxiliary head evaluated on the stricter content-disjoint folds, tracks the attended envelope better than the backward model on 6 of eight datasets (Table 2). Whether the sensor or MEG-MASC’s clean synthetic speech raises tracking we cannot say; the lag control (82.3% with pre-stimulus data) suggests the stimulus contributes.

Figure 4. Prop. 3 predicts κ\kappa from per-sample tracking. Each point is one dataset and task (linear decoder); Δζ\Delta\zeta is the Fisher-zz margin between the attended and the negative candidate’s correlation with the reconstruction. One shared constant ff fits EEG and MEG, competing and match–mismatch alike.

Flow rule and per-source residuals. Table 3. The prior-free PMI rule beats conditional likelihood by 5–25 points on every dataset, while the audio prior alone is near chance (47.0–50.9%), as Prop. 2 predicts. Conditioned on the brain, the attended talker’s flow residual is lower than the competing talker’s (ΔRcond>0\Delta R_{\mathrm{cond}} > 0 on all datasets), while the unconditional residuals are nearly equal (∣ΔRunc∣≤0.005|\Delta R_{\mathrm{unc}}| \leq0.005 on all datasets but KU Leuven, -0.012): the attention effect lies mainly in the brain-conditioned term, carried by the delta and theta bands of the envelope (App. F.5).

DatasetPMIlog p(a∣x)p(a|x)log p(a)p(a)nullΔRcond\Delta R_{\mathrm{cond}}ΔRunc\Delta R_{\mathrm{unc}}
KU Leuven69.851.847.050.3+0.003-0.012
DTU72.552.649.950.0+0.015-0.002
Fuglsang’20 (NH+HI)78.157.850.950.0+0.024-0.005
ESAA (zh)73.959.150.249.8+0.021+0.001
KUL-255ch66.855.450.149.0+0.016+0.001
Broderick57.452.049.950.7+0.006-0.001
SparrKULee85.260.150.250.2+0.039+0.000
MEG-MASC (MEG)97.177.249.749.3+0.158-0.000

Table 3.

Four talkers in a room (MAESTRO). MAESTRO ([25]) is harder in every respect. There are six loudspeakers in a room, four of them attendable; the EEG has 32 channels; listeners move freely. We follow the protocol of the concurrent MERIT study ([4]): 4-way matching over the attendable talkers with quantile-matched candidates (chance 25%), one decision per trial or per window, within listeners (content-disjoint folds) and on unseen listeners (one listener and 20% of the content held out). Table 4. Under the release’s timing, which MERIT and the benchmark use, CoupleFlow reaches 78.3% per trial within listeners (+53.6) and 67.1% on unseen listeners (+42.2). It is above MERIT’s EEG-alone model and above the benchmark in both protocols. With the playback latency corrected, so that the window truly follows the sound, it reaches 71.2% and 56.9%. This causal result is our headline number. The same-talker controls stay above chance (causal, 5-s K=2: 68.2%; K=3: 53.2%), and every 5-s null lies in [24.8, 25.3]%. Synchronisation (App. C). The release’s timestamps precede the sound by 0.78 s; we add this latency, confirmed without labels. Without it, windows cover EEG from before the sound and per-trial accuracy rises (seed 0: 77.9%, vs. 78.0% for a correctly aligned model given pre-sound EEG): our lag rule excludes that EEG by design.

Model5-s windowper trialcontribution
within listeners
CoupleFlow, release timing50.1±0.878.3±1.0+53.6±0.0
MERIT (EEG alone), release timing–69.2+39.7
Benchmark (EEG, 30 s), release timing–59.1+32.0
CoupleFlow, latency-corrected (causal)46.4±0.471.2±1.1+45.7±1.2
unseen listeners (LOSO)
CoupleFlow, release timing42.5±0.367.1±0.8+42.2±4.7
MERIT (EEG alone), release timing–60.0+31.3
Benchmark (EEG, 30 s), release timing–61.9+36.3
CoupleFlow, latency-corrected (causal)39.7±0.856.9±2.0+30.9±3.0

Table 4.

Recent side classifiers. DARN et ([44]), DBPNet (Ni et al., 2024) and ListenNet ([16]) classify the attended side from EEG alone and report 95–97% at 1 s on KU Leuven. We ran the authors’ models in our training pipeline (Table 5). Under their

Figure 3. Accumulation-law κ\kappa per dataset. Bars: 64 canonical channels, MEG-MASC 208 sensors (comp.: attended vs. competing; mm: match–mismatch). Dotted: the κ\kappa needed for 95% at 5 s.

Table 3: Flow-head decision rules at 5 s (%): prior-free PMI, conditional likelihood only, audio prior only, and the PMI rule’s EEG-shuffle null. ΔR=R(comp.)−R(att.)\Delta R = R(\mathrm{comp.}) - R(\mathrm{att.}) for the conditional (RcondR_{\mathrm{cond}}) and unconditional (Runc)R_{\mathrm{unc})} residuals; for SparrKULee and MEG-MASC the negative is the mismatched window. Seed 0. NH/HI: normal-hearing/hearing-impaired listeners; zh: Mandarin.

Table 4: MAESTRO, 4-way matched, EEG only (chance 25%). CoupleFlow: mean±s.d. over 3 seeds. Release timing: audio anchored at the earliest playback timestamp, as in MERIT and the benchmark, whose reported numbers are listed. Latency-corrected: our causal alignment. Contribution: per-trial accuracy minus the EEG-shuffle null.

within-trial split (the last 10% of each trial held out for testing) they reproduce the published numbers on KU Leuven (96.8–98.1% at 1 s). On content-disjoint folds (subject-dependent, as in the papers) they fall to 40–62% on DTU, ESAA and the 255-channel set and stays flat from 1 to 5 s: the signature of a classifier keyed to trial-specific signal, not to attention. Fuglsang’20 is the exception (67–71%), possibly alpha lateralisation or uncontrolled gaze [40]. On KU Leuven the side task is ill-posed under content-disjoint folds (one class per listener in each test fold).

Leakage control (App. § 12.§ 2). A trial split that shares stories across folds inflates none of the models by more than +2.0 points on KU Leuven and DTU; splitting windows within a trial inflated our earlier decoders by 16.7–31.5 points (Table 17), because test windows then neighbour training EEG of the same trial and label.

Ablations (App. § 12.§ 1). The per-listener correction is the one component that carries accuracy (3.0, 4.2 and 2.0 points on KU Leuven, DTU and MEG); removing any other changes 5-s accuracy there by at most 0.8 points (one seed per variant; on MAESTRO, InfoNCE also matters, App. § 12.§ 3). Reading EEG from before the sound costs 13.7–18.8 points.

Discussion and Limitations

Leakage-free evaluation lowers reported AAD accuracies, but the remaining signal is real and time-locked across nine datasets. The accumulation law turns the 5-s target many applications ask for into a statement about the signal-to-noise ratio (SNR) of the recording: EEG would need κ≈0.74\kappa\approx0.74, which only our MEG data reach (κ=1.02\kappa= 1.02). The decoder transfers to unseen listeners at a cost of 3.4–9.2 points (App. § 12.6), and sequential decisions on calibrated 1-s scores gain 5.3–7.9 points at the same mean decision time (App. § 13). On MAESTRO, where the attended talker is always the loudest, removing any one safeguard leaves every null at chance (App. Table 18): the time-centred score closes that shortcut by construction. Limitations. Prop. 2 assumes the recording depends on the attended stream only; unattended speech is weakly tracked, which the matched and same-talker sets partly address. Two datasets use published envelopes; AV-GC (no content-disjoint split) and NJU (no time-locking) are excluded; the 255-channel set has two content folds; MEG is evaluated by match–mismatch only.

AI use statement

We used a generative AI coding assistant (Claude, Anthropic) to write and run code for data processing, models, baselines and experiments, to produce tables and figures, and to draft parts of the text and proofs. It was not used to collect data, and no reported number was entered by hand: all results are generated by scripts from logged runs. The authors set the research questions and evaluation protocol, verified code with tests and data checks, checked all proofs and references, and take full responsibility for the content.

Extended Related Work

Neural basis. Cortical activity phase-locks to the slow (1–8 Hz) envelope of speech. With two competing talkers the attended talker’s envelope is represented more strongly, in MEG [15], intracranial recordings [32] and single-trial EEG [37]. The ignored talker is still weakly represented, most clearly where it is masked [8]. This is the premise of every envelope-based decoder, and the weak representation of the ignored talker is why the attended-only assumption of Prop. 2 is an approximation. We measure its strength γ\gamma on every dataset (Table 20).

Datasetrattr_{\mathrm{att}}rcompr_{\mathrm{comp}}γ\gammapred. 1−γ1-\gammaobs. linearobs. κcomp/κmm\kappa_{\mathrm{comp}}/\kappa_{\mathrm{mm}} CoupleFlow
KU Leuven0.0990.0270.280.720.710.71
DTU0.1300.0230.180.820.840.79
Fuglsang’200.1710.0180.100.900.850.83
ESAA0.1390.0190.140.860.860.98
KUL-2550.1010.0180.180.820.850.77
Broderick0.0730.0030.050.950.971.11
SparrKULee0.132–––––
MEG-MASC0.278–––––

Table 20.

Stimulus reconstruction. Backward (decoding) models map time-lagged EEG to the envelope by ridge regression [37, 10], or project EEG and audio into a shared space by canonical correlation analysis (CCA) [12]. The talker whose envelope correlates better with the reconstruction is chosen. These methods are interpretable, need little data and transfer across listeners, and they cannot use spatial or identity shortcuts. Their weakness is speed: one reconstructed correlation is a noisy statistic, and high accuracy needs tens of seconds [22]. Prop. 3 explains why: accuracy grows with T\sqrt{T} at a rate set by the per-sample tracking margin. Deep reconstructions such as VLAAI [3] raise the reconstructed correlation modestly. Under our content-disjoint folds VLAAI stays below the per-listener linear decoder on every two-talker dataset (Table 1), plausibly because its subject-independent training cannot use the listener-specific channel weights that the linear model fits. CoupleFlow keeps the time-locked principle of reconstruction but learns the neural and acoustic representations jointly, with a per-listener correction; this is the component that carries its accuracy (Table 14).

VariantKU LeuvenDTUMEG-MASC (MEG)
full model72.7±0.272.7\pm0.275.3±0.375.3\pm0.397.6±0.197.6\pm0.1
lag window from 0 ms+0.3+0.3+1.2+1.2+1.5+1.5
no flow head+0.8+0.8+0.2+0.2+0.0+0.0
no listener correction−3.0-3.0−4.2-4.2−2.0-2.0
no matched candidates+0.7+0.7−0.4-0.4+0.1+0.1
no glimpse features−0.1-0.1+0.7+0.7+0.2+0.2
no audio adversary−0.4-0.4+0.5+0.5+0.1+0.1
no null hinge+0.7+0.7−0.3-0.3+0.0+0.0
no InfoNCE−0.6-0.6−0.4-0.4−0.2-0.2
select on accuracy+0.4+0.4−0.0-0.0+0.1+0.1
no side head+0.6+0.6−0.7-0.7−0.1-0.1
no env/VICReg aux.+0.1+0.1+0.1+0.1−0.1-0.1
lag control (EEG before sound)−18.8-18.8−13.7-13.7−15.3-15.3

Table 14.

Match–mismatch. The match–mismatch task asks whether a speech segment is the one that evoked a given EEG segment [2, 39]. It is free of spatial confounds and applies to single-talker data. It is usually evaluated on unseen time windows of seen stimuli, and its negatives are often taken from nearby in the same recording. We use it on held-out stimuli as our same-talker candidate set and as the task for single-talker datasets.

Direct classifiers of the attended side. Convolutional networks [41] and their successors with attention, graph and dual-branch architectures [44, 16] (Ni et al. 2024) predict whether the attended talker is left or right from EEG alone, with 95–97% at 1 s on KU Leuven. No audio is used, so any EEG signal that differs between left and right trials can be exploited. This includes eye movements toward the attended loudspeaker, lateralised muscle activity and slow trial-specific drifts. On a dataset where gaze and attention were dissociated, accuracy followed gaze [40]. Under their within-trial split we reproduce the published numbers. On held-out stimuli, the same code falls to 40–62% on DTU, ESAA and the 255-channel set (Table 16). On KU Leuven, attended side and story are confounded, so the side task is ill-posed there without shared stories.

Datasetmodeltheir split 1 stheir split 2 stheir split 5 scontent-disj., subj.-dep. 1 scontent-disj., subj.-dep. 2 scontent-disj., subj.-dep. 5 scontent-disj., subj.-indep. 1 scontent-disj., subj.-indep. 2 scontent-disj., subj.-indep. 5 s
KU LeuvenDARNet96.896.998.052.353.153.854.258.558.9
DBPNet97.197.797.946.845.545.051.653.853.4
ListenNet98.197.897.041.943.944.055.155.757.4
DTUDARNet67.768.067.554.553.253.249.249.947.3
DBPNet66.567.668.850.852.251.447.748.248.2
ListenNet74.574.470.854.155.055.249.148.149.6
ESAADARNet85.485.786.056.058.659.455.854.954.5
DBPNet87.889.689.159.861.661.655.856.858.1
ListenNet91.691.288.556.558.859.556.057.959.3
Fuglsang’20DARNet82.782.882.867.168.768.965.765.367.8
DBPNet83.683.681.868.368.868.666.868.469.7
ListenNet88.687.685.470.571.271.368.971.071.2
KUL-255DARNet98.598.999.041.940.940.244.641.844.4
DBPNet99.499.699.640.843.342.343.141.942.1
ListenNet98.698.498.147.146.545.648.447.148.8

Table 16. Recent side classifiers (authors’ models in our training pipeline), accuracy (%) under their within-trial split (last 10% of each trial held out for testing; earlier windows split 9:1 into training and validation) and under our content-disjoint folds, subject-dependent and subject-independent. On KU Leuven the content-disjoint side task is ill-posed (one class per listener in each test fold).

Brain decoding with learned representations and generative models. Contrastive alignment between M/EEG and pretrained speech representations identifies which of many speech segments was heard [14]. Flow matching [31] and diffusion models give tractable conditional generative models, and conditioning dropout lets one network represent both p(a∣x)p(a \mid x) and p(a)p(a) (Ho and Salimans, 2021). Likelihood-ratio classification with generative models is classical. We use the ratio for a different purpose: it cancels the audio prior p(a)p(a), which would otherwise favour acoustically typical candidates regardless of the brain (Prop. 2; `)Table 3. Our InfoNCE term follows contrastive decoding, restricted to windows of one listener so that identity cannot solve it.

Evaluation validity. Data leakage through non-independent splits inflates results across machine-learning (ML) based science [28]. In AAD, reviews have noted within-trial splits and repeated stimuli [39], and Rotaru et al. (2024) demonstrated gaze and trial-identity confounds. Table 6 lists the failure modes we found or controlled, and the control we use for each.

Failure modeEffectControl in this work
Within-trial window splittest windows neighbour training windows of the same trial; any trial-specific signal is rewardedcontent-disjoint folds (Prop. 4); 86.4%→55.9% for our earlier decoder (Table 17)
Repeated stimulithe same story in training and test, across or within listenerscontent groups recovered from the audio; trial-split control (Table 15)
Gaze and lateralised artefactsside classifiers read eye position rather than attentionno side information in the decision path; side head reported separately
Audio-only cuesloudness or spectral differences mark the attended talker (MAESTRO: always loudest)time-centred score (Prop. 1); quantile-matched candidates; adversary; Table 18
Trial identity and slow stateEEG identifies the trial, not the attended talkeracross-trial and circular-shift nulls
Wrong timingthe “post-stimulus” window contains pre-stimulus EEG, or there is no time-locking at alloffset scans on every bundle; MAESTRO latency correction; NJU excluded (App. § 9)
Selection on test datahyperparameters tuned on the test foldlag window and checkpoints chosen on validation only

Table 6. Failure modes in AAD evaluation, what they inflate, and the control in this work.

Multi-talker rooms. MAESTRO [25] records EEG while listeners attend one of several loudspeakers in a room, and the concurrent MERIT [4] evaluates multimodal decoders on it. We follow MERIT’s 4-way protocol and show that the release timing places the decision window partly before the sound (§ 4, App. § 9).

Proofs

Proof of Proposition 4. Suppose a test stream of trial ii were linked to a training or validation stream of trial jj. Then GG has an edge (i,j)(i,j), so ii and jj lie in the same connected component. Test folds are unions of whole components and training and validation data come only from the other components, a contradiction. Content shared only in chunks shorter than 10 s or below the threshold is not covered.

Proof of Proposition 1. Condition on yy. By (b), xx is independent of a1:Ka_{1:K} given yy, so by (c) the law of (x,aπ(1),…,aπ(K))(x,a_{\pi(1)},\ldots,a_{\pi(K)}) given yy is invariant to every permutation π\pi, and by (a) so is the law of (s1,…,sK)=(g(x,a1),…,g(x,aK))(s_1,\ldots,s_K)=(g(x,a_1),\ldots,g(x,a_K)). Each index is then the unique maximiser with the same probability, (1−P(tie∣y))/K(1-P(\mathrm{tie}\mid y))/K; counting ties as errors, P(y^=y∣y)≤1/KP(\hat{y}=y\mid y)\leq1/K, and averaging over yy gives the claim. For the time-centred score (Eq. (1)) a time-constant embedding yields sk=0s_k=0 for all kk, a tie, hence an error: the acoustic pathway alone cannot exceed chance. Assumption (c) holds exactly for the same-talker set (the negative is the attended talker’s own speech) and approximately for the competing set in counterbalanced designs; the across-trial and circular nulls instantiate (b).

Proof of Proposition 2. By Bayes’ rule, (a) and (b), P(y=k∣x,a1:K)∝p(x∣a1:K,y=k)P(y=k∣a1:K)=p(x∣ak)/KP(y=k\mid x,a_{1:K})\propto p(x\mid a_{1:K},y=k)P(y=k\mid a_{1:K})=p(x\mid a_k)/K. Bayes’ rule for the pair (ay,x)(a_y,x) gives p(x∣a)=p(a∣x)p(x)/p(a)p(x\mid a)=p(a\mid x)p(x)/p(a), evaluated at a=aka=a_k. The factor p(x)p(x) is common to all kk, so log⁡P(y=k∣⋅)=PMI⁡(ak;x)+const\log P(y=k\mid\cdot)=\operatorname{PMI}(a_k;x)+\mathrm{const} and the MAP decision maximises PMI. Replacing PMI by log⁡p(ak∣x)\log p(a_k\mid x) adds log⁡p(ak)\log p(a_k) to candidate kk’s score, which depends on the audio only.

Proof of Proposition 3. By (i) and (ii), zj∼N(ζj,1/(neff−3))z_j\sim\mathcal{N}(\zeta_j,1/(n_{\mathrm{eff}}-3)) with neff=fTn_{\mathrm{eff}}=fT, and by (iii) zatt−zneg∼N(Δζ,2/(neff−3))z_{\mathrm{att}}-z_{\mathrm{neg}}\sim\mathcal{N}(\Delta\zeta,2/(n_{\mathrm{eff}}-3)). Since atanh⁡\operatorname{atanh} is increasing, the decision is correct iff zatt>znegz_{\mathrm{att}}>z_{\mathrm{neg}}, so P(correct∣T)=Φ(Δζ(neff−3)/2)P(\mathrm{correct}\mid T)=\Phi(\Delta\zeta\sqrt{(n_{\mathrm{eff}}-3)/2}). For neff≫3n_{\mathrm{eff}}\gg3 this is Φ(κT)\Phi(\kappa\sqrt{T}) with κ=Δζf/2\kappa=\Delta\zeta\sqrt{f/2}; solving Φ(κTp)=p\Phi(\kappa\sqrt{T_p})=p gives TpT_p. For the noise model, write a~=v+η\tilde{a}=v+\eta with η\eta independent of vv and of the envelopes, and σ=Var⁡(v)/Var⁡(η)\sigma=\operatorname{Var}(v)/\operatorname{Var}(\eta). Then Cov⁡(a,a~j)=Cov⁡(v,aj)\operatorname{Cov}(a,\tilde{a}_j)=\operatorname{Cov}(v,a_j) and Var⁡(a~)=Var⁡(v)(1+σ)/σ\operatorname{Var}(\tilde{a})=\operatorname{Var}(v)(1+\sigma)/\sigma, so ρj=ρj∗σ/(1+σ)\rho_j=\rho_j^*\sqrt{\sigma/(1+\sigma)} with ρj∗=corr⁡(v,aj)\rho_j^*=\operatorname{corr}(v,a_j). For small ρj\rho_j, atanh⁡ρj≈ρj\operatorname{atanh}\rho_j\approx\rho_j, so Δζ≈(ρatt∗−ρneg∗)σ/(1+σ)\Delta\zeta\approx(\rho_{\mathrm{att}}^*-\rho_{\mathrm{neg}}^*)\sqrt{\sigma/(1+\sigma)} and κ2∝σ/(1+σ)\kappa^2\propto\sigma/(1+\sigma). Multiplying σ\sigma by a factor b>0b>0 multiplies κ2\kappa^2 by b(1+σ)/(1+bσ)b(1+\sigma)/(1+b\sigma): linear at low SNR, saturating at high SNR. Assumption (iii) is exact when the two envelopes are uncorrelated and approximate when they are weakly correlated.

Datasets and Preprocessing

Table 7: Datasets after unified preprocessing. Groups: content groups recovered from the audio; folds = min(5, groups). Env.: gammatone envelope computed from the audio (ours) or published by the dataset authors (prov.). HI: hearing-impaired. 1-/2-talker: single talker or two competing talkers.

Datasets. All data are public releases, recorded by their authors with ethics approval and informed consent; we collected no new data. Table 8 lists every release, the version we used and its license. “Our use” states what we changed from the release.

Datasetrepositoryidentifierversion usedlicenseaccess
KU LeuvenZenodo10.5281/zenodo.4004271v2.0 (2019)1CC BY-NC-SA 4.0open
DTUZenodo10.5281/zenodo.1199011v1 (2018)CC BY-NC 4.0open
Fuglsang’20Zenodo10.5281/zenodo.36182052020 (BIDS 1.3.0)CC BY 4.0open (audio on request)
ESAAZenodo10.5281/zenodo.7795585v1.0 (2022)CC BY 4.0open
KUL-255chZenodo10.5281/zenodo.4518754v1.1.0 (2021)CC BY-NC 4.0open
BroderickDryad10.5061/dryad.070jcv3 (2019)CC0 1.0open
SparrKULeeKU Leuven RDR10.48804/K3VSNDv3.1 (2023)CC BY-NC 4.0open; 107 recordings on request
MEG-MASCOSF10.17605/OSF.IO/AG3KJ2022CC0 1.0open
MAESTROHugging Faceaspire-osu/maestro-eeg-dataset2026CC BY-NC-SA 4.0open
NJU (excl.)Zenodo10.5281/zenodo.7253438v1 (2022)CC BY 4.0open
AV-GC (excl.)Zenodo10.5281/zenodo.11058711v1 (2024)CC BY-SA 4.0open (13 of 16 listeners)

Table 8. Releases, versions and licenses (checked on the release pages, September 2026). CC: Creative Commons; BY: attribution; NC: non-commercial; SA: share-alike; CC0: public-domain dedication. BIDS: Brain Imaging Data Structure; RDR: Research Data Repository; OSF: Open Science Framework. Excluded datasets are listed for completeness.

KU Leuven ([11, 6]). 16 normal-hearing listeners, 64-channel BioSemi ActiveTwo, released at 128 Hz. Four Dutch short stories read by male narrators, played at 60 dB(A) (A-weighted sound level) through insert phones, either dichotically (dry) or filtered with head-related transfer functions (HRTFs) to ±90°. 20 trials per listener (8 of about 6 min and 12 two-minute repetitions, about 72 min); the attended ear is fixed within a trial. Our use: all 20 trials, envelopes from the dry story recordings, content groups = the four stories.

DTU ([18]). 18 normal-hearing listeners, 64-channel BioSemi at 512 Hz. Danish fictional stories read by a male and a female professional speaker, about 50 s per trial, 60 trials per listener, talkers at ±60∘\pm60^\circ through simulated room impulse responses (anechoic, mildly and highly reverberant rooms) and insert earphones; the target talker is cued before each trial. Our use: the two-talker trials only (single-talker trials and one trial that stops after 19.6 s are dropped).

Fuglsang’20 ([19]). 44 listeners aged 51–76, 22 with sensorineural hearing loss and 22 with normal hearing; 64-channel BioSemi at 512 Hz. Two audiobooks read by a male and a female talker at ±90∘\pm90^\circ (non-individualised HRTFs, insert phones), 48 trials of about 50 s, of which 32 are two-talker; the competing talker starts 3–5 s after the target. The release contains no audio, only the authors’ envelopes. Our use: the two-talker trials (32 per listener; 25 for one listener) and the provided envelopes of the unamplified stimuli (hearing-impaired listeners heard an amplified version), so every listener shares one envelope per stimulus.

ESAA ((Li et al., 2022)). The release holds 17 normal-hearing listeners (the paper reports 20), 64-channel EEG recorded at 1 kHz (BrainProducts files). Mandarin short stories told by a female and a male professional storyteller, HRTF-filtered to ±90∘\pm90^\circ, 32 trials of 55–90 s per listener (one recording ends after 16 s). Our use: all trials, including those the authors flag for poor comprehension answers; the stimulus reaches the ear about 0.3 s after the start marker, which we correct (App. § 9, alignment checks).

KUL-255ch ([35]). 30 normal-hearing male listeners aged 22–35, 255-electrode 10-5 cap (253 EEG channels in the files), Compumedics SynAmps RT at 1 kHz. Two parts of two of the KU Leuven Dutch stories, HRTF-filtered to the left and right, four trials of about 6.5 min, one per attended ear and story part. Our use: all four trials. Because the stories are those of KU Leuven, the two datasets are never mixed across training and test. The authors corrected their stimulus table in August 2026; our attended labels are checked on the data, as the labelled attended envelope is the one the EEG tracks (Table 1).

Broderick ([9]). 33 native English listeners (34 recorded), 128-channel BioSemi ActiveTwo plus mastoids at 512 Hz, released at 128 Hz. Two Jules Verne novels read by two male speakers, one per ear over headphones (Twenty Thousand Leagues Under the Sea left, Journey to the Centre of the Earth right), 30 trials of about 60 s; each listener attends the same side in every trial (listeners 1–17 left, 18–33 right). The release contains envelopes, not audio. Our use: the “Cocktail Party” part, the provided envelopes, and the 64 channels nearest the canonical layout (all 128 in Table 10).

Datasetchannels5 strial
Broderick64 / 12862.3 / 66.481.5 / 86.5
KUL-255ch64 / 25367.1 / 69.998.3 / 100.0

Table 10. Canonical 64 vs. all native channels (5-s and trial accuracy, %; 125-ms lag window, seed 0).

SparrKULee ([1]). 85 normal-hearing native Flemish listeners, 64-channel BioSemi ActiveTwo (recorded at 8192 Hz, distributed at 1024 Hz). Single-talker Flemish audiobooks and podcasts, 6–10 trials of about 15 min per listener. Our use: the recordings of the training set of the auditory-EEG challenge at the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); we drop the stories that also occur in KU Leuven and the two speech-in-noise runs, and re-time the audio to the EEG clock with the recorded trigger pulses.

MEG-MASC ([24]). 27 English speakers, 208-channel KIT axial gradiometer MEG at 1 kHz. Four synthesised English fictional stories (about 2 h, one or two sessions), interleaved with short word lists, with comprehension questions. Our use: every played audio file as one trial, sound onsets corrected by the delay measured on the recorded audio channel, and all 208 sensors, read by a 208-channel input layer. A per-listener principal component analysis (PCA) to 64 components, which we used first, fits different components for each listener; it matched the 208-sensor model on the content folds but cannot transfer to an unseen listener (54.1% instead of 95.0% under leave-one-listener-out evaluation), so no model in the paper uses it.

MAESTRO ([25]). 16 listeners aged 18–30, 32-channel ANT Neuro eego at 500 Hz, plus eye gaze, pupil, egocentric video and head motion (not used here). English speech from six loudspeakers in a room: four attendable talkers and two distractors, with background noise; 100 evaluation trials of about 30 s and 5 familiarisation trials. Our use: the EEG of the 100 evaluation trials, with the synchronisation described below.

Excluded. NJU ([45]) and AV-GC ([40]), for the reasons given below; their licenses are in Table 8.

Licenses. Every license permits our use, which is non-commercial research. We redistribute no data, stimuli or derived features; the released code contains no data and reads the original files, so the share-alike terms (KU Leuven, MAESTRO) do not apply to it. SparrKULee’s stimuli are licensed for non-commercial use only, and 107 of its recordings are available only on request; we used the recordings distributed for the ICASSP 2024 challenge.

Content groups. Table 9 summarises the groups behind the content-disjoint folds (§ 4). Holding out trials does not hold out audio: most datasets play the same stories to all listeners, and KU Leuven replays each story pair five times per listener, so a trial split puts the same speech in training and test, where a decoder with an audio pathway can memorise it. The audio, not the metadata, defines the groups because stimulus keys are missing, inconsistent or per-listener in the releases (e.g. Fuglsang’20 numbers its files per listener, yet its 1401 trials contain only 4 distinct contents). The search has two stages. First, every candidate envelope is averaged to 8 Hz, cut into non-overlapping 10-s chunks, and each chunk is correlated with every 10-s window, at every 1/8-s offset, of every other envelope; pairs with r>0.9r > 0.9 are candidates. Second, each candidate is re-correlated at 64 Hz over the full overlap at the implied offset (±1/8 s) and kept if r>0.95r > 0.95. A copy of one recording gives r≈1r \approx1 (every kept link has r≥0.964r \ge0.964). Different contents stay well below the threshold: the 64-Hz correlation at the best 8-Hz match of a 10-s chunk in another group has a median of at most 0.53 per dataset (one listener’s chunks against all envelopes of a second listener). The closest cases (Table 9, last column), up to 0.93, have identifiable causes that are not shared speech: in Fuglsang’20 every competing-talker envelope starts with 3–5 s of zero padding, and in SparrKULee one 10-s stretch of a podcast resembles stretches of three other podcasts. Two envelopes are linked if both stages pass. Let G\mathcal{G} be the graph whose nodes are trials, with an edge between two trials whenever any of their envelopes are linked; its connected components, found with union–find, are the content groups (e.g. the 320 KU Leuven trials form 4 groups).

Datasettrialsgroupsgroups per listenersingle-trial groupsnumbermin. rr (64 Hz)max. rr (8 Hz)max. rr (64 Hz)
KU Leuven3204403161.0000.710.60
DTU10796058–59010190.9880.880.86
Fuglsang’20140144013970.9770.940.88
ESAA544323205121.0000.740.57
KUL-255ch1202201181.0000.690.57
Broderick9843029–3009541.0000.770.66
SparrKULee550632–904870.9640.940.93
MEG-MASC14703030014401.0000.820.67
MAESTRO1600100100015001.0000.720.59

Table 9. Content groups recovered from the audio. Groups per listener: how many groups one listener heard. Single-trial groups: content heard in one trial only. Links: 64-Hz-confirmed links that merged two components (trials − groups). Last two columns: best 8-Hz match of a 10-s chunk in another group, and the 64-Hz correlation at that location: the chance level the 0.95 threshold must exceed.

Proposition 4 (content disjointness). No test stream is linked to any training or validation stream.

The proof is immediate from the component construction (App. § 8). Folds are unions of whole groups (min(5, groups) folds); validation uses whole training groups when at least 10 remain, otherwise the last 15% of each training trial of at least 200 s (Fuglsang’20, whose trials last about 50 s: the last 20% of each trial of at least 30 s), separated from the training part by a gap of one context margin on each side. Such validation data share content with training and serve only for early stopping and the lag choice; no reported number uses them.

Alignment checks. For every bundle, a per-listener ridge backward model (lags 0–250 ms, leave-trials-out) tracks the attended envelope (held-out r=0.07r = 0.07–0.27, against r≈0r \approx0 for the envelope shifted by 1–4 s), and a global offset scan (125-ms steps) peaks at 0 s, or at +125 ms on KU Leuven and Broderick. For SparrKULee the audio was re-timed to the EEG clock using the recorded 1-Hz trigger pulses (drift 3.4–5.5 × 10−510^{-5}). For MEG-MASC, sound onsets were corrected by the delay measured on the audio channel recorded in the MEG file (40–56 ms).

Preprocessing details. The steps, in order: bad-channel removal, average reference, resampling to 128 Hz, notch, 0.5–32 Hz band-pass, mapping to 64 canonical channels by name or, for high-density caps, nearest position (missing channels are zero; MEG keeps its 208 sensors), resampling to 64 Hz, and a z-score per trial and channel. Bad channels: standard deviation outside 0.2–5 times the median channel’s (MAESTRO: the release’s list, flat channels, and channels at the clipping level in at least 1% of samples); they are zeroed. Reference: mean of the good channels after removing each channel’s mean (MEG is not re-referenced; its bad sensors are the union over a listener’s runs). Notch: second-order IIR at the mains frequency (Q=30Q = 30). Band-pass: fourth-order Butterworth, 0.5–32 Hz, applied forwards and backwards (zero phase). Envelope: mean over 28 gammatone bands (ERB-spaced, 50 Hz–5 kHz) of the compressed magnitude ∣⋅∣0.6|\cdot|^{0.6}. The published envelopes differ: Fuglsang’20 provides 24 gammatone bands, 100–4000 Hz, ∣⋅∣0.3|\cdot|^{0.3}, of the unamplified stimuli; Broderick provides its own broadband envelope. SparrKULee stories that overlap the KU Leuven stimuli were removed.

MAESTRO synchronisation. Each stream is trimmed to the release’s common window, from the latest start among EEG and audio playback to the earliest end. The playback start is known only from software timestamps. The glasses’ scene microphone recorded the room audio, and a label-free search by generalised cross-correlation with phase transform (GCC-PHAT, 300–4000 Hz) located every talker’s audio in it. On 1,240 trials with clear peaks, the three playback devices start within a median 14 ms of each other (90th percentile 118 ms), while their timestamps differ by a median 162 ms. Slicing each device from its own timestamp therefore adds misalignment; it lowers tracking. We place all talkers at the earliest timestamp plus one global latency of 0.781 s (50 samples at 64 Hz). Odd and even listeners give the same estimate (0.75 and 0.81 s). EEG tracking of the six-talker mixture, which uses no attention label, peaks at 0.000 s after the correction. There is no per-trial or per-listener adjustment. Without the correction, a window that nominally follows the sound in fact covers EEG from 0.78 s before it, and per-trial accuracy rises to 77.9%. A correctly aligned model given the same pre-sound EEG reaches the same 78.0% (vs. 70.2% causal), so the extra accuracy comes from EEG that precedes the sound. It is consistent with the lag control: pre-sound EEG alone gives 46.3% per trial on MAESTRO. The attended talker is always the loudest, by 3–18 dB, but per-trial accuracy does not depend on that margin (Spearman ρ=0.03\rho= 0.03, p=0.22p = 0.22). Bad channels are those listed in the release, plus flat channels and channels at the amplifier’s clipping level in at least 1% of samples. M1/M2 are dropped by the 64-channel mapping, leaving 28.0 channels on average.

Excluded datasets. NJU [45] (Mandarin, 32-channel): no attended-envelope tracking at any global offset (±60 s), time stretch (0.8–1.25), trial pairing, or alignment to the recorded playback audio. A decoder trained on KU Leuven, which transfers to DTU (r=0.030r = 0.030), gives r=0.005r = 0.005 on NJU. The released audio is spliced from the playback recording with removed stretches, so its timing cannot be restored. AV-GC [40]: its podcasts are re-paired across listeners, so all trials form one content component and no content-disjoint split exists.

DECAF. Trained with the same folds and selection rule, the recursive decoder reached test r≈0r \approx0 on the two-talker sets (also its EEG branch alone) and only weak tracking on the single-talker sets (Table 13); on Fuglsang’20 it falls below chance, and in 3 of 33 folds validation selected the untrained model. We therefore do not report it as a baseline.

Baseline training details. Linear: per-listener ridge regression from EEG at lags 0–250 ms to the attended envelope; penalty from {10−2,…,102}\{10^{-2},\ldots,10^{2}\} times tr⁡(XTX)/D\operatorname{tr}(X^{\mathsf{T}}X)/D (DD: number of lagged features) chosen on the listener’s validation data, then refit on training and validation data. Where whole-group validation holds none of a listener’s trials (137 SparrKULee listener-folds), the penalty is chosen by leave-one-segment-out cross-validation on that listener’s training segments. VLAAI and Dil. MM are trained across all training listeners of a fold: 5-s windows with 80% overlap, Adam (learning rate 10−310^{-3}), batch 64, early stopping on validation (patience 5), at most 1,500 steps per epoch and 95 minutes; VLAAI minimises −r-r, Dil. MM the cross-entropy against a mismatched window of the same trial. All baselines use CoupleFlow’s splits, validation data and test windows. On MEG-MASC they read all 208 sensors; VLAAI’s output-context layer then has 208 instead of 64 filters, so that its skip connection matches the input width.

Training details. AdamW (learning rate 10−310^{-3}, weight decay 10−410^{-4}), gradient-norm clipping at 1, batches of up to 32 windows from one listener (a listener’s last partial batch is kept if it has at least 4), 5-s training crops, up to 100 epochs with early stopping (patience 20) on validation contribution, learning rate halved when it has not improved for more than 5 epochs. Nulls. The across-trial null uses 20 permutations at 5 s and 3 at other windows; when a listener has a single test trial, windows of that trial are permuted instead. The circular null (5 s only) uses 5 random shifts of at least 2 s. Shapley values split the accuracy above the null between the two EEG pathways (coupling and side head). Validation contribution is measured on the matched candidates (same-talker on SparrKULee and MEG-MASC) with 3 across-trial permutations. Loss weights: side 0.3, InfoNCE 1, null hinge 0.5 (margin 0.5), VICReg 0.1, adversary 0.3, flow 0.3, envelope 0.2; they were set once and are identical in every reported run. What differs between datasets: the lag window (chosen on validation), the validation tails of Fuglsang’20, and MEG-MASC’s 208-channel input. Development. The input features (8- and 28-band sub-bands and glimpses) and the training length were chosen during development on KU Leuven fold 0, whose test accuracy we inspected; no other dataset or fold was used for design choices. Without fold 0, KU Leuven scores 72.8% at 5 s (all folds: 72.7%). Each run used one NVIDIA A100 graphics processing unit (GPU); a fold takes 5–215 minutes including test evaluation.

Network Details

Table 11 lists every module with its parameter count, read from the trained checkpoints. All datasets share one architecture and one set of hyperparameters, except the lag window and Fuglsang’20’s validation tails (App. C). Only the number of listeners, and hence of listener corrections, and the width of the audio input change. The input has 54 channels per candidate with sub-band features and 4 for the two datasets with published envelopes only. The full model has 126,607–148,879 parameters per dataset. The part that makes the decision, listener correction, virtual sensors, both encoders and the score, has only 25,689 shared parameters plus 256 per listener. LL is the number of samples in the decision window (320 at 5 s) and L′=L+200L^\prime= L + 200 includes 100 context samples on each side.

ModuleSpecificationOutputParameters
Listener correctionx+UmVmTxx + U_mV_m^{\mathsf{T}}x, rank 2, UmU_m init. 064×L′64 \times L^{\prime}256 / listener
Virtual sensors1×11 \times1 conv, 64 →\to 88×L′8 \times L^{\prime}520
EEG encoder5 conv, kernel 3, dilation 1–16, width 32, GN+GELU, no padding16×L16 \times L11,952
Audio encoder (per kk)4 conv, kernel 3, dilation 1–8, width 32, causal, shared16×L16 \times L13,200 (8,400*)
Coupling scoreweights wdw_d, temperature τ\tauKK17
Flow head1×11 \times1 in (34 →\to 64), 5 residual dilated blocks, zero-init1×L1 \times L85,505
Side head1×11 \times1 (64 →\to 16), 4 dilated conv, mean/log-s.d., linear210,659
Audio adversaryMLP 32 →\to 32 →\to 1 per candidate, gradient reversalKK1,089
Envelope read-out1×11 \times1 conv, 16 →\to 11×L1 \times L17
Shared total122,959
Decision path (coupling)listener correction, virtual sensors, both encoders, score25,689 + 256/listener

Table 11. CoupleFlow modules and parameter counts (KU Leuven checkpoint; * envelope-only input). GN: GroupNorm; GELU: Gaussian error linear unit; MLP: multilayer perceptron. Only the rows in the decision path are needed at test time for the coupling decision; the flow head is needed for the PMI rule.

Input and listener correction. Every EEG dataset is mapped to the same 64 canonical channels (missing channels are zero), so one network serves all caps. MEG-MASC keeps its 208 sensors, in the same order for every listener (one helmet), and is the only dataset with a different input layer: the listener correction, the virtual sensors and the side head’s first layer take 208 instead of 64 channels; nothing else changes. Listeners differ in head shape, cap placement and source orientation. A full per-listener spatial filter (64×6464 \times64) would overfit the few minutes of data per listener, so we add a rank-2 correction (Figure 5) x+UmVmTxx + U_m V_m^{\mathsf{T}}x for listener m=1,…,Mm = 1,\ldots,M (MM: number of listeners in the dataset), with 256 parameters per listener. UmU_m starts at zero, so training starts from one shared model and each listener departs from it only as far as its data support. This is the component that carries accuracy: removing it lowers 5-s accuracy by 3.0, 4.2 and 2.0 points on KU Leuven, DTU and MEG-MASC (Table 14) and changes MAESTRO per-trial accuracy by -8.7 points (Table 18).

Input, listener correction and virtual sensors

Figure 5. Input, listener correction and virtual sensors. The correction reads two spatial patterns of the EEG with VmV_m and writes them back with UmU_m; the identity path keeps the shared model, and Um=0U_m = 0 at initialisation, so a listener departs from it only as far as the data require. Bottom: maps from the KU Leuven fold-0 checkpoint, namely VmV_m and UmU_m of the listener with the largest correction, and the activation patterns ΣWT(WΣWT)−1\Sigma W^{\mathsf{T}}(W\Sigma W^{\mathsf{T}})^{-1} of the eight virtual sensors, with WW their weights and Σ\Sigma the covariance of the corrected training EEG [26]. Diverging colour scale per map, grey at zero.

Variantfoldsunmatched (5s) acc.unmatched (5s) contrib.matched acc.matched nullmatched contrib.matched circ. nulltrialtrial contrib.collapse
full model545.6+20.645.925.0+20.925.070.2+45.40.00
no flow head546.1+21.045.925.1+20.825.070.1+46.30.00
no listener correction541.6+16.541.725.1+16.525.161.4+36.40.00
unmatched training545.5+20.545.825.0+20.825.169.9+44.70.00
no glimpse features544.9+19.945.025.1+20.025.168.8+43.80.00
no sub-band features544.3+19.344.525.0+19.525.168.3+42.80.00
no audio adversary545.8+20.745.925.1+20.825.170.0+45.30.00
no null hinge545.4+20.445.325.0+20.325.170.9+46.70.00
no InfoNCE543.3+18.243.325.1+18.225.166.6+42.10.00
select on accuracy545.7+20.845.524.9+20.625.070.3+46.20.00
no side head545.2+20.345.325.0+20.325.270.3+46.10.00
no env/ VICReg aux.545.2+20.245.125.0+20.024.968.9+44.30.00
lag control (EEG before sound)533.6+8.533.525.1+8.425.246.3+20.80.00

Table 18.

Virtual sensors. A 1×11 \times1 convolution maps the 64 channels to 8 virtual sensors, the learned analogue of the spatial filters of backward models and CCA [10, 12]. The narrow bottleneck forces the encoder to use a few spatial patterns, which suits the low spatial resolution of EEG and limits overfitting. Their activation patterns (Figure 5, bottom) are smooth and spatially broad.

EEG encoder and lag window. Five convolutions (Figure 6) with kernel 3 and dilations 1, 2, 4, 8, 16 give a receptive field of 63 samples (0.98 s at 64 Hz). This covers the latencies at which temporal convolution functions find cortical responses to speech, with room for slower components. The convolutions are unpadded, so no output depends on zero padding, and the output is sliced so that ete_t depends only on EEG from t+δt + \delta to t+δ+0.98t + \delta+ 0.98 s after the sound at tt (δ∈{0,125}\delta\in\{0,125\} ms, chosen on validation). EEG before the sound cannot be a response to it, so this window excludes anticipation and slow drifts shared with the past. The mirror window, EEG before the sound, is our lag control. GroupNorm (4 groups) is used because batches are small and come from one listener. The last layer has no activation, so the embedding is not restricted to positive values, and dropout 0.1 follows every hidden layer.

Receptive fields of both encoders on one time axis

Figure 6. Receptive fields of both encoders on one time axis, drawn to scale (one dot per sample at 64 Hz; each node depends on three nodes of the layer below, spaced by the dilation dd). The neural embedding ete_t reads EEG from δ\delta to δ+0.98\delta+ 0.98 s after the sound at tt (blue; δ=125\delta= 125 ms shown, 0 ms on six datasets, chosen on validation), and the candidate embedding a^k,t\hat{a}_{k,t} reads the 0.48 s of audio up to tt (orange). Matching the two therefore fixes the lag direction: the brain response must follow the sound. The lag control uses the mirrored window before the sound (hatched). Each decision window carries 100 samples (1.56 s) of context on both sides, more than either receptive field needs, so no output depends on zero padding.

Candidate features and quantile matching. Each candidate is described by its broadband gammatone envelope (∣⋅∣0.6|\cdot|^{0.6}), its onsets (half-wave-rectified derivative, since cortex responds strongly to acoustic edges), 8- and 28-band sub-band envelopes (spectral detail), and glimpsed and masked channels: the candidate’s window-standardised sub-band envelope where it exceeds, or falls below, the largest standardised envelope of the other candidates (on the broadband envelope for the two envelope-only datasets). Ignored speech is tracked mainly where it is glimpsed ([8]), so these channels let the model treat the two cases differently. In training, and for the matched and same-talker candidate sets, the broadband envelope and onset of each window (with its context) are quantile-matched across candidates; every other channel is z-scored within the window. The competing set is evaluated unmatched. The candidates then have identical envelope value distributions, which removes loudness and level cues that would identify the attended talker without the brain (Figure 7).

Candidate features on one 5-s KU Leuven test window

Figure 7. Candidate features on one 5-s KU Leuven test window. (a) The broadband envelope before and after quantile matching, its onset channel, the 8-band sub-band envelopes and the glimpsed and masked parts of one band. (b) How the 54 input channels of each candidate are formed; the 8- and 28-band envelopes enter as they are, and glimpses are computed on the 8-band envelopes. (c) Matching gives both candidates the same value distribution over the window while each keeps its own time course.

Audio encoder. Four causal convolutions (Figure 6, orange) (dilations 1–8, receptive field 31 samples, 0.48 s) map each candidate to a^k∈R16×L\hat{a}_k \in\mathbb{R}^{16 \times L}. Causal means a^k,t\hat{a}_{k,t} depends only on audio up to tt, so together with the EEG window the lag direction is fixed by the architecture: EEG must follow the sound it is matched to. One set of weights is shared by all candidates. This is required by Prop. 1 (assumption (a)) and means no candidate can be favoured by its position.

Coupling score. The score (Eq. (1)) is a weighted sum of per-dimension Pearson correlations over time, scaled by a learned temperature (initialised at 0.07) and centred over candidates. Correlation is invariant to the scale and offset of either embedding, so the score depends only on how the two signals co-vary in time. Centring the embeddings over time makes the score exactly zero for a time-constant neural embedding. This is what makes the decoder null-calibrated: without time-varying neural input there is nothing for the audio pathway to use. The weights ωd\omega_d start equal (1/D1/D) and learn which embedding dimensions carry coupling. Figure 8 follows one test window through the score.

The time-centred coupling score on one KU Leuven test window

Figure 8. The time-centred coupling score on one KU Leuven test window (fold-0 model). The EEG embedding and both candidate embeddings (heat maps, each dimension standardised) are centred and normalised over time; each dimension gives one correlation per candidate (bars: attended in orange, competing in grey), the learned weights ωd\omega_d combine them, and centring over candidates yields the scores, which pick the attended talker here. Inset: without time-varying neural input every score is zero.

Flow head. The flow head models the attended envelope a∈RLa \in\mathbb{R}^{L} (the z-scored broadband envelope channel of one window) given the neural embedding. Flow matching joins Gaussian noise ϵ\epsilon to the data along the straight path a(u)=(1−u) ε+u aa^{(u)} = (1-u)\,\varepsilon+ u\,a, indexed by the flow time u∈[0,1]u \in[0,1] (ε∼N(0,IL)\varepsilon\sim\mathcal{N}(0,I_L)), whose velocity is a−εa-\varepsilon, and trains a network vθ(a(u),u,c)v_\theta(a^{(u)},u,c) with parameters θ\theta and condition cc to predict that velocity:

LFM=Eu∼U[0,1], ε[∥vθ(a(u),u,c)−(a−ε)∥2].(3)\mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{u\sim\mathcal{U}[0,1],\,\varepsilon}\left[\left\|v_\theta(a^{(u)},u,c) - (a-\varepsilon)\right\|^2\right]. \tag*{(3)}

The condition is c=ec=e; with probability 0.2 it is set to zero and a flag marks it as absent (written c=∅c=\varnothing), as in classifier-free guidance (Ho and Salimans 2021). Integrating vθv_\theta from u=0u=0 to 11 carries noise to envelopes, so one network defines both p(a∣x)p(a \mid x) (with c=ec=e) and p(a)p(a) (with c=∅c=\varnothing). A candidate is scored by how well each model explains it: its residuals Rcond(ak)=Eu,ε∥vθ(ak(u),u,e)−(ak−ε)∥2R_{\mathrm{cond}}(a_k)=\mathbb{E}_{u,\varepsilon}\lVert v_\theta(a_k^{(u)},u,e)-(a_k-\varepsilon)\rVert^2 and Runc(ak)R_{\mathrm{unc}}(a_k), the same with c=∅c=\varnothing. A lower residual means a more typical sample, since the expected flow-matching residual acts as a (weighted) surrogate for the negative log-likelihood. The flow score Runc(ak)−Rcond(ak)R_{\mathrm{unc}}(a_k)-R_{\mathrm{cond}}(a_k), how much the recording lowers the error on aka_k, is therefore a proxy for the pointwise mutual information PMI⁡(ak;x)=log⁡p(ak∣x)−log⁡p(ak)\operatorname{PMI}(a_k;x)=\log p(a_k\mid x)-\log p(a_k), which is the Bayes-optimal score (Prop. 2). The audio-only term log⁡p(ak)\log p(a_k) cancels, and a network that ignores ee scores every candidate 0, as Eq. (1) does. We also report the per-source residuals of the attended and the competing talker.

The network’s input at each time step concatenates the noisy envelope a(u)a^{(u)} (1 channel), ee (16), a flag indicating whether the condition is present (1) and a sinusoidal embedding of flow time (16). A 1×1 convolution maps these 34 channels to 64, followed by five residual blocks (dilated convolution 1–16 with same padding, GroupNorm, GELU, 1×1 convolution) and a zero-initialised output layer, so training starts from zero velocity. At test time the residuals Rcond,RuncR_{\mathrm{cond}},R_{\mathrm{unc}} are averaged over 8 Monte-Carlo draws of flow time (stratified) and noise. The draws are shared across candidates and across the conditional and unconditional passes, so their noise cancels in Runc−RcondR_{\mathrm{unc}}-R_{\mathrm{cond}}. It is the largest module (85,505 parameters) but lies outside the coupling path. Removing it changes coupling accuracy by at most 0.8 points (Table 14). We keep it because it supplies the PMI rule and the per-source residuals (Table 3). Figure 9 shows both uses.

Flow head

Figure 9. Flow head. (a) Training: the velocity network sees the noisy envelope a(u)a^{(u)} on the straight path from noise ε\varepsilon to the attended envelope aa, the neural embedding (dropped with probability 0.2), a flag saying whether it is present, and the flow time uu, and regresses the path velocity a−εa-\varepsilon. (b) Test: every candidate is scored with and without the neural embedding on the same Monte-Carlo draws; the difference of the residuals estimates its pointwise mutual information with the recording (Prop. 2).

Side head. A separate EEG-only branch (Figure 10a) (1×1 convolution to 16 channels, four dilated convolutions, mean and log standard deviation over time, linear layer) predicts the attended side. When the candidates’ sides are known, its log-probability of each candidate’s side is centred over candidates and scaled by a learned gain (softplus(gsideg_{\mathrm{side}}), gsideg_{\mathrm{side}} initialised at −2). During training this side score is added to sks_k inside LCE\mathcal{L}_{\mathrm{CE}}. All reported accuracies use the coupling score alone, and the side head’s contribution is reported separately. The branch gives lateralised EEG a separate, measurable route instead of letting the coupling path absorb it. Removing it changes coupling accuracy by at most 0.8 points.

Auxiliary heads: side head, audio-only adversary, and envelope read-out

Figure 10. Auxiliary heads. (a) The side head reads only the listener-corrected EEG; its side log-probabilities enter the training loss through the candidate scores but never the reported accuracies, which use the coupling score alone. (b) The audio-only adversary tries to find the attended candidate from the audio embeddings; the gradient-reversal layer (GRL) passes its gradient to the audio encoder with the sign flipped, so the encoder removes any such cue. (c) The envelope read-out reports reconstruction accuracy and is not used by any decision rule.

Audio adversary and envelope read-out. The adversary (Figure 10b) summarises each candidate embedding by its temporal mean and standard deviation (32 values) and scores it with a two-layer MLP. A softmax over candidates gives Ladv\mathcal{L}_{\mathrm{adv}}, and a gradient-reversal layer (coefficient 1) makes the audio encoder remove any feature that identifies the attended talker from audio alone (Ganin et al., [20]). The envelope read-out, a 1×1 convolution from ee, reports how well the neural embedding reconstructs the attended envelope; it does not enter the decision (Figure 10c).

Size and compute. The decision path has 25,689 shared parameters, a size that suits wearable hardware and the limited data per listener. All modules are convolutional and run in time linear in the window length. Training runs on one GPU in minutes to a few hours per fold (App. § 9).

Training Objective

The objective (Eq. (2)) has one term that asks for the decision and several that shape how the decision is reached. This section explains each term’s role in plain terms, how the weights were set, and how large each term is in practice (Table 12).

DatasetepochLCE\mathcal{L}_{\mathrm{CE}}LNCE\mathcal{L}_{\mathrm{NCE}}Lnull\mathcal{L}_{\mathrm{null}}LVIC\mathcal{L}_{\mathrm{VIC}}Ladv\mathcal{L}_{\mathrm{adv}}LFM\mathcal{L}_{\mathrm{FM}}Lside\mathcal{L}_{\mathrm{side}}Lenv\mathcal{L}_{\mathrm{env}}
KU Leuvenfirst0.673.360.900.000.210.290.200.19
KU Leuvenselected0.402.050.270.000.210.260.120.17
DTUfirst0.663.200.820.000.210.290.210.19
DTUselected0.431.780.210.000.210.230.190.17
Fuglsang’20first0.653.150.870.000.210.300.200.19
Fuglsang’20selected0.401.870.260.000.210.260.150.16
ESAAfirst0.683.320.930.000.210.310.210.19
ESAAselected0.391.970.250.000.210.250.150.17
KUL-255chfirst0.693.100.930.000.210.310.210.20
KUL-255chselected0.381.900.290.000.210.250.090.17
Broderickfirst0.713.350.920.000.210.300.220.20
Broderickselected0.272.350.350.000.210.260.000.18
SparrKULeefirst0.532.790.570.000.210.27–0.18
SparrKULeeselected0.381.370.140.000.210.26–0.17
MEG-MASCfirst0.421.750.270.000.210.27–0.16
MEG-MASCselected0.320.290.010.000.210.23–0.12
chance0.693.47––0.21–0.21–

Table 12. Weighted loss terms during training (each already multiplied by its weight; mean over folds and seeds of the logged epoch average), at the first epoch and at the selected epoch. Chance: value of a term for a decoder at chance (K=2K = 2 candidates, batch B=32B = 32). “–”: the side head needs two talkers in known positions. Five Broderick runs evaluated from saved checkpoints have no logged history and are omitted.

Definitions. A training batch holds B≤32B \le32 windows i=1,…,Bi=1,\ldots,B, all from one listener. Window ii has the recording x(i)x^{(i)}, the neural embedding e(i)e^{(i)}, the acoustic embeddings a^k(i)\hat{a}^{(i)}_k of its KK candidates, the attended index yiy_i and the attended envelope a(i)=ayia^{(i)}=a_{y_i}. We write sik=s(e(i),a^k(i))s_{ik}=s(e^{(i)},\hat{a}^{(i)}_k) for the score of Eq. (1) before centring and sˉik\bar{s}_{ik} for the centred score sks_k of window ii. The terms of Eq. (2) are as follows, with sp⁡(z)=log⁡(1+ez)\operatorname{sp}(z)=\log(1+e^z) the softplus function.

  • Task. LCE=−1B∑i∑kqiklog⁡πik\mathcal{L}_{\mathrm{CE}}=-\frac{1}{B}\sum_i\sum_k q_{ik}\log\pi_{ik}, the cross-entropy (CE) between the predicted distribution πik=softmax⁡k(sˉik+hik)\pi_{ik}=\operatorname{softmax}_k(\bar{s}_{ik}+h_{ik}) and the label qik=0.9 [k=yi]+0.1/Kq_{ik}=0.9\,[k=y_i]+0.1/K ([k=yi][k=y_i] is 1 if k=yik=y_i, else 0), smoothed because the label is the instruction, not what the listener did at every moment (attention lapses). hikh_{ik} is the side head’s score for candidate kk (see Lside\mathcal{L}_{\mathrm{side}}; 0 when the candidates’ sides are unknown); it is used only in training, and every reported decision uses sˉik\bar{s}_{ik} alone.

  • Contrastive matching. With Gij=s(e(i),a^yj(j))G_{ij}=s(e^{(i)},\hat{a}^{(j)}_{y_j}), the score between window ii’s brain response and window jj’s attended speech, LNCE=−12B∑i[log⁡eGii∑jeGij+log⁡eGii∑jeGji]\mathcal{L}_{\mathrm{NCE}}=-\frac{1}{2B}\sum_i\left[\log\frac{e^{G_{ii}}}{\sum_j e^{G_{ij}}}+\log\frac{e^{G_{ii}}}{\sum_j e^{G_{ji}}}\right] (symmetric InfoNCE). It asks which of the BB speech segments evoked each response, and vice versa: a 32-way problem, which gives the encoders far more signal per window than the 2-way task. Since all windows come from one listener, only the response’s time course, not who is listening, can solve it.

  • Null hinge. Lnull=1B∑i[sp⁡(s(e(i−1),a^yi(i))−si,yi+ξ)+sp⁡(ξ−si,yi)]\mathcal{L}_{\mathrm{null}}=\frac{1}{B}\sum_i\left[\operatorname{sp}(s(e^{(i-1)},\hat{a}^{(i)}_{y_i})-s_{i,y_i}+\xi)+\operatorname{sp}(\xi-s_{i,y_i})\right], margin ξ=0.5\xi=0.5, e(0)=e(B)e^{(0)}=e^{(B)}. The attended speech must score higher with its own recording than with another window’s recording (first term) and than with a constant recording, which scores 0 (second term). LCE\mathcal{L}_{\mathrm{CE}} only compares candidates within a window; this term ties the score to the recording itself, a training-time version of the null test.

  • Audio-only adversary. A small network reads only the acoustic embeddings: the time mean and standard deviation (s.d.) of each a^k(i)\hat{a}^{(i)}_k (32 numbers) pass through a two-layer perceptron to one logit per candidate, and Ladv\mathcal{L}_{\mathrm{adv}} is the cross-entropy of its softmax against yiy_i. The adversary minimises Ladv\mathcal{L}_{\mathrm{adv}}; between it and the acoustic encoder sits a gradient-reversal layer (Ganin et al. [20]), which flips the sign of the gradient, so the encoder is trained to make the adversary fail. This removes any acoustic cue (loudness, voice) that reveals the attended talker without the brain.

  • Collapse guard. LVIC=116B∑i,d[sdmin⁡−sdt(ed(i))]++0.0416∑d≠d′Cdd′2\mathcal{L}_{\mathrm{VIC}} = \frac{1}{16B}\sum_{i,d}\left[\mathrm{sd}_{\min}-\mathrm{sd}_{t}(e^{(i)}_d)\right]_{+}+\frac{0.04}{16}\sum_{d\ne d'}C_{dd'}^{2}, with sdt\mathrm{sd}_{t} the s.d. over time, sdmin⁡=0.5\mathrm{sd}_{\min}=0.5, [z]+=max⁡(z,0)[z]_{+}=\max(z,0), and CC the 16×1616\times16 covariance of the time-centred embedding over all samples of the batch: the variance and covariance terms of VICReg (Bardes et al. [5]). A constant embedding scores every candidate 0 and stops learning; copies of one dimension waste the other 15. The term stays negligible unless either starts.

  • Flow matching. LFM\mathcal{L}_{\mathrm{FM}} is Eq. (3) with a=a(i)a=a^{(i)} and c=e(i)c=e^{(i)} (or c=∅c=\varnothing), averaged over the batch with one draw of (u,ϵ)(u,\epsilon) per window. It trains the flow head; through cc it also asks e(i)e^{(i)} to carry what predicts the attended envelope.

  • Side head. A separate EEG-only network (App. § 10) gives probabilities pside(left∣x(i))p_{\mathrm{side}}(\mathrm{left}\mid x^{(i)}) and pside(right∣x(i))p_{\mathrm{side}}(\mathrm{right}\mid x^{(i)}) that the attended talker is on that side, and Lside=−1∣B∣∑i∈Blog⁡pside(zi∣x(i))\mathcal{L}_{\mathrm{side}}=-\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\log p_{\mathrm{side}}(z_i\mid x^{(i)}), with ziz_i the attended side and B\mathcal{B} the windows where it is known. hikh_{ik} is this log-probability for candidate kk’s side, centred over candidates and scaled by a learned gain. Lateralised EEG can carry spatial attention, but also gaze or listener identity; a separate head lets us measure its contribution and keep it out of reported decisions.

  • Envelope read-out. Lenv=1B∑i[1−r(ψ⊤e(i),a(i))]\mathcal{L}_{\mathrm{env}}=\frac{1}{B}\sum_i\left[1-r(\psi^\top e^{(i)},a^{(i)})\right], with ψ∈R16\psi\in\mathbb{R}^{16} a learned linear map and rr the Pearson correlation over time. It reports how well the neural embedding reconstructs the attended envelope (Table 2), comparable to backward models, and is not used in decisions.

The weights, λNCE=1\lambda_{\mathrm{NCE}}=1, λnull=0.5\lambda_{\mathrm{null}}=0.5, λadv=λFM=λside=0.3\lambda_{\mathrm{adv}}=\lambda_{\mathrm{FM}}=\lambda_{\mathrm{side}}=0.3, λVIC=0.1\lambda_{\mathrm{VIC}}=0.1, λenv=0.2\lambda_{\mathrm{env}}=0.2, were set once and are the same for every dataset: the two decoding terms at full weight, the others below them so that the decision shapes the embedding.

Why more than one term. Cross-entropy over the candidates is all the task requires, but on its own it accepts any route to the label. AAD data offer several routes that do not go through the brain response: the attended talker may be louder or have a recognisable voice, every window of a trial has the same label, and a listener may attend the same side throughout. A decoder that takes such a route scores well on a leaky split and fails on new content. The architecture already closes the main route: the time-centred score is zero for every candidate without time-varying neural input (Prop. 1). The remaining terms make the useful route easier to find (InfoNCE), guard against degenerate solutions (null hinge, VICReg, adversary), and add outputs that the decision does not use but that we report (flow head, side head, envelope read-out).

Decoding terms. LCE\mathcal{L}_{\mathrm{CE}} is the task: given one window and its candidates, put the highest score on the attended one. Label smoothing (0.1) reflects that the label is an instruction, not an observation: listeners lapse, so a fraction of windows is mislabelled and should not be fitted with full confidence. With two candidates LCE\mathcal{L}_{\mathrm{CE}} gives one bit of supervision per window, too little to teach two encoders from scratch. LNCE\mathcal{L}_{\mathrm{NCE}} supplies more: it asks which of the B=32B=32 windows’ attended speech goes with which window’s recording, a BB-way matching problem (ln 32 = 3.47 nats at chance against ln 2 = 0.69 for the task). Because all windows in a batch come from one listener, recognising the listener cannot solve it; only the time course of the response can. It is the largest term at the first epoch (Table 12) and falls by 30–83% as the encoders learn to align brain and speech. Both terms act on the same score (Eq. (1)), so what InfoNCE learns transfers directly to the decision.

Anti-shortcut terms. Lnull\mathcal{L}_{\mathrm{null}} states the null hypothesis as a constraint: the true recording must give the attended speech a higher score than the recording of another window, and than a constant recording (score 0), each by a margin ξ=0.5\xi=0.5. A decoder that scores the attended talker high whatever the brain did violates it. The softplus keeps a small gradient after the margin is met. Ladv\mathcal{L}_{\mathrm{adv}} trains an audio-only classifier to find the attended candidate from summary statistics of the candidate embeddings; the reversed gradient (Ganin et al. [20]) then removes from the acoustic encoder whatever let it succeed. Its value is a direct check: it is 0.3 ln 2 = 0.21, chance, at the first and at the selected epoch on every dataset, so after quantile matching the audio carries no usable cue to the attended role. LVIC\mathcal{L}_{\mathrm{VIC}} is a guard against a collapsed neural embedding (constant over time, or with all dimensions alike), which would score every candidate 0 and stall training. Its variance part is zero while every dimension’s temporal s.d. exceeds sdmin⁡=0.5\mathrm{sd}_{\min} = 0.5; the whole term is below 0.005 at the first and at the selected epoch on every dataset. These three terms are therefore insurance: they cost nothing when the model behaves and act only when it starts to take a shortcut. On MAESTRO, where a loudness shortcut exists, removing any one of them leaves every null at chance (Table 18), because the score already closes the shortcut.

Density and auxiliary terms. LFM\mathcal{L}_{\mathrm{FM}} (Eq. 3) trains the flow head, which supplies the PMI rule and the per-source residuals (Table 3). Its gradient also reaches the neural encoder through the condition, and asks the embedding to carry what predicts the attended envelope, which agrees with the coupling objective. Lside\mathcal{L}_{\mathrm{side}} trains the EEG-only side head. Lateralised EEG can carry attention, but also gaze and, when a listener attends one side throughout, listener identity: on Broderick the term drops to 0 because side and listener coincide. We therefore measure its credit separately and never use it in a reported decision. Lenv=1−r\mathcal{L}_{\mathrm{env}} = 1 - r trains a linear read-out of the embedding to the attended envelope; it gives the tracking values of Table 2 and slightly regularises the embedding towards the envelope.

How the weights were set. The weights were set once and not tuned on any dataset; every dataset uses the same values. Where the concurrent MERIT objective [4] has a corresponding term we took its weight; the others follow from scale. The task has weight 1 and is the reference. InfoNCE also has weight 1: it is the same log-softmax on the same score, and should dominate early. The null hinge (0.5) is a constraint that should bind only when violated. The adversary (0.3) is kept moderate because a gradient-reversal game becomes unstable when the reversed gradient dominates; the flow, side and envelope terms (0.3, 0.3, 0.2) are kept below the decoding terms so that the neural embedding is shaped mainly by the decision; VICReg (0.1) is a guard. Two facts make the choice uncritical. First, the selected model is chosen on validation contribution (accuracy minus the across-trial null), not on the loss, so the weights trade off training signals, not the selection criterion. Second, removing any one term changes 5-s accuracy on KU Leuven, DTU and MEG-MASC by at most 0.8 points (Table 14; one seed per variant); only the per-listener correction, an architectural component, matters there. On MAESTRO, InfoNCE does matter (-3.6 points per trial, Table 18).

Table 12: Weighted loss terms during training (each already multiplied by its weight; mean over folds and seeds of the logged epoch average), at the first epoch and at the selected epoch. Chance: value of a term for a decoder at chance (K=2K = 2 candidates, batch B=32B = 32). “–”: the side head needs two talkers in known positions. Five Broderick runs evaluated from saved checkpoints have no logged history and are omitted.

Additional Results

Table 13 gives CoupleFlow and every baseline at every decision window. Sec. F.1 removes one component at a time, Sec. F.2 compares two kinds of leakage, Sec. F.3 repeats the ablation on MAESTRO, and Sec. F.6 holds out listeners. Table 19 lists CoupleFlow variants, and Table 20 tests the accumulation law (Prop. 3).

Ablations

Table 14, 5-s accuracy, all folds, relative to the 125-ms model with 3 seeds; each variant is one run (seed 0). The per-listener low-rank correction is the component that carries accuracy: without it the model loses 3.0 (KU Leuven), 4.2 (DTU) and 2.0 points (MEG). Starting the lag window at 0 ms adds +0.3, +1.2 and +1.5. Every other component changes 5-s accuracy by at most 0.8 points (one seed per variant). This includes the flow head, matched and glimpsed candidates, the adversary, the null hinge, InfoNCE, contribution-based selection and the side head. These components are therefore not accuracy devices. The flow head supplies the PMI decision rule and the per-source residuals (Table 3); the adversary, null hinge and selection rule are safeguards, whose effect is on the null and credit diagnostics rather than on accuracy. The lag control, which reads neural data from 1.1 s to 125 ms before the sound, loses 18.8, 13.7 and 15.3 points (53.8, 61.6 and 82.3%). MEG remains well above chance in this control. We attribute this, as a hypothesis, to the regular rhythm of its synthetic speech, which makes the envelope predictable from the preceding response.

Table 14: Change in 5-s accuracy (points) when one component is removed; full model: mean±s.d. over 3 seeds, 125-ms lag window; variants: seed 0.

Two kinds of leakage

Table 15 and Table 17 can look contradictory: one finds that a leaky split inflates accuracy by at most +2.0 points, the other by 16.7–31.5. They measure different leaks.

Datasetmodeltrial splitcontent-disj.inflation
KU LeuvenLinear66.264.9+1.2
VLAAI63.563.8-0.2
Dilated MM66.565.9+0.6
CoupleFlow74.872.8+2.0
DTULinear72.272.4-0.3
VLAAI64.664.8-0.2
Dilated MM69.069.1-0.1
CoupleFlow75.575.4+0.1

Table 15. Content leakage: 5-s competing accuracy (%) when folds share stories (trial split: whole trials held out, but the stories of test trials also occur in training trials) against content-disjoint folds. Every model is retrained on each split. CoupleFlow: seed 0, 125-ms lag window on both splits.

Datasetlistenersearlier flow decoderearlier envelope decoder
within-trialtrial-disj.dropwithin-trialtrial-disj.drop
KU Leuven1686.455.930.576.157.518.6
DTU1883.656.227.474.457.716.7
NJUa^{a}2186.054.531.570.150.719.4

Table 17.

Content leakage (Table 15). In a trial split, whole trials are held out, so no test sample, and no EEG from a test trial, is seen in training. Only the stories repeat: a test trial’s audio was heard by the listener in another, training trial. A model can profit from this only if it can recognise the audio and map it to a label, as an audio-only or stimulus-identity classifier could. To measure what a content-sharing split does, we retrain every model on KU Leuven and DTU with one group per trial, the split used in most prior work, so test stories also occur in training (Table 15). No model gains more than +2.0 points: -0.3 to +2.0 for the linear decoder, VLAAI, the match–mismatch model and CoupleFlow alike. Sharing stories across trials is therefore not what inflates these decoders. Each of them reaches the decision through the neural response to the candidate audio, with no path that can memorise a story’s identity. The small KU Leuven gain for CoupleFlow (+2.0) is consistent with its trial split exposing all four story pairs in training instead of three.

Trial-identity leakage (Table 17). In a within-trial split, every trial contributes windows to both training and test: our earlier decoders used a random 5-fold split of overlapping windows, the side classifiers’ protocol holds out the last 10% of each trial. Each test window then has training windows from the same trial, with the same label, overlapping it or seconds away. EEG has slow, trial-specific structure: electrode impedances, drifts, posture and gaze, and an attended side that is constant within the trial. A decoder that recognises which trial a window comes from therefore knows its label without decoding attention. This is the much larger effect: 16.7–31.5 points for our earlier decoders, and 96.8–98.1% at 1 s on KU Leuven for the side classifiers (Table 16); on DTU our runs under their split come out about 12 points below the published numbers, plausibly because of our stricter preprocessing: 0.5-Hz high-pass, per-trial z-scoring, no electrooculography (EOG) channels. NJU, whose EEG is not time-locked to the audio at all, reached 86.0% this way. The two tables therefore agree: what inflates AAD accuracy is evaluating on windows of trials seen in training, not on stories heard in training. Content-disjoint folds rule out both.

Table 17: Our earlier decoders (a flow decoder and an envelope decoder) under two splits, 5-s windows, one model per listener, mean over listeners (%). Within-trial: 5-fold split of randomly assigned windows with 50% overlap; test windows neighbour training windows of the same trial. Trial-disjoint: whole trials held out, no shared samples. Everything else (data, listeners, recipe, folds) is identical. Trial-disjoint is still not content-disjoint on KU Leuven, whose stories repeat across trials. a^{a}NJU shows no EEG–audio time-locking (App. § 9), so its within-trial accuracy can only come from the split.

Ablation on MAESTRO

Table 18 repeats the ablation on MAESTRO, where the audio itself marks the attended talker. Two results stand out. First, there is no discrepancy between accuracy and contribution for any variant. Unmatched and matched candidates give the same accuracy to within 0.3 points, and every null stays at chance. Removing the adversary, the null hinge, contribution-based selection or matched training (each changes per-trial accuracy by at most 0.8 points) does not open the shortcut. It is the architecture that closes it: the time-centred coupling score gives the acoustic pathway no route to the decision (Prop. 1). The safeguards are therefore redundant with it rather than necessary. Second, accuracy is carried by the components that shape the brain–speech coupling. Per-trial accuracy changes by -8.7 points without the per-listener correction, -3.6 without InfoNCE, -1.9 without sub-band features, -1.4 without glimpse channels and -1.3 without the auxiliary losses.

Table 18: Ablation on MAESTRO, where the audio offers a shortcut (the attended talker is always the loudest): within listeners, seed 0, 4-way. For each variant: 5-s accuracy and contribution with unmatched candidates (shortcut available) and with matched candidates; the EEG-shuffle and circular-shift nulls; per-trial accuracy and contribution; and embedding collapse (mean cosine between windows; 1 = every window alike).

Variants and the accumulation law

Task and sensor. For the linear decoder the task prediction κcomp/κmm=1−γ\kappa_{\mathrm{comp}}/\kappa_{\mathrm{mm}} = 1-\gamma has mean error 0.02 (r=0.95r = 0.95, p=0.004p = 0.004), and for CoupleFlow 0.07 (r=0.84r = 0.84); the task factor 1/(1−γ)1/(1-\gamma) is 1.05–1.38 (mean 1.19). The law κ=f/Δ\kappa= \sqrt{f}/\Delta with f=9.7f = 9.7 fits the linear values with mean error 3.7%. At matched task and listening condition (single talker, match–mismatch), MEG’s per-sample tracking is 2.10× that of SparrKULee’s EEG, which predicts a κ\kappa ratio of 2.14; we observe 2.03 for the linear decoder and 1.62 for CoupleFlow, which is compressed by its ceiling near 99%. Single-talker listening is not what raises EEG tracking: SparrKULee’s attended correlation (0.13) lies inside the two-talker range (0.07–0.17). CoupleFlow’s read-out reaches 0.082–0.148 on the two-talker EEG sets and 0.396 on MEG.

Table 19 lists CoupleFlow variants at 5 s, and Table 20 gives the tracking values behind the test of Prop. 3 in § 4.

Dataset125 ms (base)0 msenv. onlyrank 8listener FTnative ch.
KU Leuven72.873.072.173.473.2n/a
DTU75.476.474.475.675.1n/a
Fuglsang’2079.979.5n/a––n/a
ESAA71.277.972.172.071.4n/a
KUL-25567.169.267.0––69.9
Broderick62.363.5n/a––66.4
SparrKULee86.692.385.3––n/a
MEG-MASC97.699.097.4––n/a

Table 19.

Table 19: CoupleFlow variants, 5-s accuracy (%, seed 0). 125 ms / 0 ms: start of the neural lag window. env. only: broadband envelope without sub-band features (Fuglsang’20 and Broderick are envelope-only in every column). rank 8: larger per-listener correction. listener FT: per-listener fine-tuning after pooled training. native ch.: all native channels instead of the 64 canonical ones (125-ms window). n/a: the variant equals the base (envelope-only releases; caps with 64 EEG channels). “–”: not run. Rank 8 and listener FT were run only on KU Leuven, DTU and ESAA, to test whether listener heterogeneity explained the ESAA gap; neither did. MEG-MASC uses its 208 sensors throughout (208-channel input layer).

Table 20: Test of Prop. 3: tracking of the attended (rattr_{\mathrm{att}}) and competing (rcompr_{\mathrm{comp}}) envelope by the per-listener linear decoder (all listeners), ignored-talker ratio γ\gamma, the predicted and the observed ratio of competing to match–mismatch κ\kappa.

Further analyses

Where the attention signal lives. Decomposing the PMI attention effect (PMIatt_{\mathrm{att}} − PMIcomp_{\mathrm{comp}}) by modulation band of the candidate envelope (seed 0), the delta band (0.5–4 Hz) carries 71–91% of it on every dataset, theta (4–8 Hz) 7–22%, and alpha and beta together at most 9%. This matches the known spectral profile of cortical speech tracking ([15]), and it matches for MEG and EEG alike.

Hearing-impaired listeners. On Fuglsang’20, 5-s accuracy is 80.6% for the 22 hearing-impaired and 78.5% for the 22 normal-hearing listeners (mean over 3 seeds; Mann–Whitney p=0.06p = 0.06–0.460.46 across seeds). Decoding is not worse with hearing loss, consistent with the enhanced envelope tracking reported for these listeners ([19]). We use the release’s envelopes of the unamplified stimuli for every listener.

Leave-one-listener-out evaluation

The content-disjoint folds hold out stories but test listeners whose other stories were seen in training. Here each fold holds out one listener and one content fold: the test set is that listener’s trials on the held-out content, and training uses only the other listeners’ trials on the other content, so neither the test listener nor the test audio is seen in training (validation: whole training groups when at least 10 remain, else trial tails, as in App. § 9). Every fold passed the same split audit as the content folds (no shared trial, content, stimulus file or sample, and no test-listener data in training). Each dataset’s paper configuration and validation-chosen lag are used unchanged, with a single seed.

Table 21. An unseen listener costs 3.4–9.2 points at 5 s on every dataset except Broderick, and accuracy keeps rising with the window (20-s column). On Broderick the decoder does not transfer to new listeners at all; a pooled linear backward model under the same folds fails in the same way (last column), so this is a property of that recording, not of the model. Where both transfer, CoupleFlow stays ahead of the pooled linear model.

Datasetcontent folds 5 sLOSO, CoupleFlow (1 seed) 1 sLOSO, CoupleFlow (1 seed) 5 sLOSO, CoupleFlow (1 seed) 20 sLOSO, CoupleFlow (1 seed) null (5 s)LOSO, linear 5 s
KU Leuven72.758.068.081.449.859.7
DTU76.258.570.484.250.064.0
Fuglsang’2079.661.073.789.050.267.5
ESAA77.958.868.683.050.466.1
KUL-255ch69.457.566.079.750.164.0
Broderick63.649.950.150.049.749.2
SparrKULee92.268.285.997.150.1–
MEG-MAS Ca99.076.195.099.750.0–

Table 21. Leave-one-listener-out (LOSO) accuracy (%), listener and content held out together, one seed. Content folds: the main-table 5-s accuracy (3 seeds) for reference. Null: EEG from another trial of the same listener (another window of the same trial when the listener has a single test trial). Linear: pooled ridge backward model on the same folds (two-talker sets; penalty chosen on the first 40 validation segments). a208-sensor input (App. § 10). MAESTRO’s LOSO results are in § 4.

Confidence and Sequential Decisions

Everything so far scores fixed windows. Here we ask how the decoder’s per-window scores should be combined over time, when a decision should be made, and whether overlapping windows or a different training window help. Nothing in this appendix retrains the paper’s models except Sec. G.6; the decision rules only read their per-window scores.

Theory

Setting and notation. A test trial is cut into consecutive, non-overlapping windows i=1,2,…i = 1, 2, \ldots of length ww seconds. Window ii has the recording xix_i and the two candidate envelopes a1,i,a2,ia_{1,i}, a_{2,i}; we write a⋅,i=(a1,i,a2,i)a_{\cdot,i} = (a_{1,i}, a_{2,i}), x1:n=(x1,…,xn)x_{1:n} = (x_1, \ldots, x_n) and a1:n=(a⋅,1,…,a⋅,n)a_{1:n} = (a_{\cdot,1}, \ldots, a_{\cdot,n}). The attended index y∈{1,2}y \in\{1, 2\} is fixed within the trial. PkP_k and EkE_k denote probability and expectation given y=ky = k (and the candidates), log is the natural logarithm, Λ\Lambda is the logistic function, Λ(⋅)=1/(1+exp⁡(−⋅))\Lambda(\cdot) = 1/(1 + \exp(-\cdot)), and Φ\Phi and ϕ\phi are the standard normal distribution and density functions. The per-window log-likelihood ratio is

ℓi=log⁡p(xi∣y=1,a⋅,i)−log⁡p(xi∣y=2,a⋅,i);\ell_i = \log p(x_i \mid y=1, a_{\cdot,i}) - \log p(x_i \mid y=2, a_{\cdot,i});

under the assumptions of Prop. 2 it equals PMI(a1,i;xi)(a_{1,i};x_i) − PMI(a2,i;xi)(a_{2,i};x_i). After nn windows the accumulated evidence is Sn=∑i=1nℓiS_n = \sum_{i=1}^{n} \ell_i (S0=0S_0 = 0). A real statistic ℓ^\hat{\ell} is calibrated when P(y=1∣ℓ^)=Λ(ℓ^)P(y=1 \mid\hat{\ell}) = \Lambda(\hat{\ell}). A decoder gives the candidates scores s1,i,s2,is_{1,i},s_{2,i} (§ 3); we calibrate their margin Δsi=s1,i−s2,i\Delta s_i = s_{1,i} - s_{2,i} by one scale, ℓ^i=βΔsi\hat{\ell}_i = \beta\Delta s_i, with β>0\beta> 0 fitted by the logistic loss on the fold’s validation windows, and use ℓ^i\hat{\ell}_i in place of ℓi\ell_i. A positive scale never changes a single-window decision; it matters only when windows are combined.

Proposition 5 (additive evidence). *Assume (a) given yy and all candidates, the xix_i are independent and xix_i depends on the candidates only through ay,ia_{y,i}; (b) P(y=1)=12P(y=1) = \frac{1}{2} independently of the candidates. Then log⁡P(y=1∣x1:n,a1:n)P(y=2∣x1:n,a1:n)=Sn\log\frac{P(y=1\mid x_{1:n},a_{1:n})}{P(y=2\mid x_{1:n},a_{1:n})} = S_n, i.e. SnS_n is calibrated, and the decision with the fewest errors given nn windows is sign⁡Sn\operatorname{sign} S_n. Majority voting over sign⁡ℓi\operatorname{sign}\ell_i is optimal only when all ∣ℓi∣|\ell_i| are equal.

Proposition 6 (accumulation law as an information rate). Let the ℓi\ell_i of Prop. 5 be independent and identically distributed (i.i.d.) given yy, with ℓi∣y=1∼N(μ,ν2)\ell_i \mid y=1 \sim\mathcal{N}(\mu,\nu^2) for some μ>0\mu> 0, ν2>0\nu^2 > 0. Then (i) ν2=2μ\nu^2 = 2\mu and ℓi∣y=2∼N(−μ,2μ)\ell_i \mid y=2 \sim\mathcal{N}(-\mu,2\mu); (ii) μ\mu is the Kullback–Leibler divergence DKL(p(xi∣y=1)∥p(xi∣y=2))D_{\mathrm{KL}}(p(x_i \mid y=1) \mathbin{\|} p(x_i \mid y=2)), the information one window carries about yy; (iii) with the information rate I=μ/wI=\mu/w (nats per second) and a fixed decision length of T=nwT=nw seconds (nn windows),

P(correct∣T)=Φ(IT/2)=Φ(κT),κ2=I/2.P(\mathrm{correct} \mid T)=\Phi(\sqrt{IT/2})=\Phi(\kappa\sqrt{T}), \qquad\kappa^2=I/2.

Prop. 6 gives Prop. 3 a decision-theoretic reading that does not rely on the decoder being a correlation: 2κ22\kappa^2 is the information the brain response carries about the attended talker, per second. It also gives a check of calibration and of the Gaussian model: by (i), exact Gaussian log-likelihood ratios have Var⁡1(ℓi)/(2E1ℓi)=1\operatorname{Var}_1(\ell_i)/(2\mathbb{E}_1\ell_i)=1, where Var⁡1\operatorname{Var}_1 is the variance under P1P_1; we evaluate it on test windows with the log-odds oriented so that the attended candidate is candidate 1.

Proposition 7 (voting wastes evidence). Under Prop. 6 let q=Φ(μ/2)q=\Phi(\sqrt{\mu/2}) be the one-window accuracy. Summing nn windows gives Φ(n Φ−1(q))\Phi(\sqrt{n}\,\Phi^{-1}(q)). Majority voting gives, for large nn, Φ(n(2q−1)/(2q(1−q)))\Phi\left(\sqrt{n}(2q-1)/(2\sqrt{q(1-q)})\right). As μ→0\mu\to0 voting needs π/2≈1.57\pi/2 \approx1.57 times as many windows as summing for the same accuracy.

Proposition 8 (sequential decisions). Under Prop. 6, fix a threshold A>0A>0 and let N=min⁡{n:∣Sn∣≥A}N=\min\{n:|S_n|\ge A\} be the number of windows the rule reads before ∣Sn∣|S_n| first reaches AA; the rule decides sign⁡SN\operatorname{sign}S_N after TA=NwT_A=Nw seconds (Wald’s sequential probability ratio test with symmetric thresholds). Neglecting the overshoot of SNS_N beyond ±A\pm A, (i) its error rate is α=1/(1+eA)\alpha=1/(1+e^A); (ii) its mean decision time is

E[TA]=(1−2α)log⁡1−αα2κ2 s;\mathbb{E}[T_A]=\frac{(1-2\alpha)\log\frac{1-\alpha}{\alpha}}{2\kappa^2}\ \mathrm{s};

(iii) no rule with both error rates ≤α\le\alpha has a smaller mean decision time under either hypothesis [43]. A fixed-length decision reaches the same error rate α\alpha after Tfix=(Φ−1(1−α)/κ)2T_{\mathrm{fix}}=(\Phi^{-1}(1-\alpha)/\kappa)^2 seconds (Prop. 6}), so E[TA]/Tfix=0.59,0.54,0.49,0.42\mathbb{E}[T_A]/T_{\mathrm{fix}}=0.59, 0.54, 0.49, 0.42 at α=0.2,0.1,0.05,0.01\alpha=0.2, 0.1, 0.05, 0.01, decreasing to 1/41/4 as α→0\alpha\to0.

The sequential rule reaches the fixed-window accuracy in about half the time, with the same κ\kappa. This is the classical result of Wald [42]; its continuous-time limit is the drift–diffusion model of perceptual decisions [7]. It rests on two assumptions that EEG violates to some degree, independence of consecutive windows and a constant information rate; we measure both below. If the information rate varies between stretches of a trial (attention lapses, noise bursts), the sequential rule spends its time on the weak stretches, which a fixed window cannot do.

Proposition 9 (stopping does not bias the posterior). Under the assumptions of Prop. 5, let NN be any stopping time of the window sequence, i.e. whether to stop after window nn depends only on x1:nx_{1:n} and a1:na_{1:n}, and decide sign⁡SN\operatorname{sign}S_N. Then P(correct∣x1:N)=Λ(∣SN∣)P(\mathrm{correct}\mid x_{1:N})=\Lambda(|S_N|). In particular every decision taken when ∣SN∣≥A|S_N|\ge A is correct with probability at least Λ(A)\Lambda(A), whatever its length, and the accuracy of the sequential rule is set by the threshold alone; the length only adapts to how fast the evidence arrives.

Proposition 10 (overlapping windows). Let the evidence be additive in time: the log-likelihood ratio of the data in [t0,t1)[t_0,t_1) is ℓ(t0,t1)=∫t0t1ℓ˙(t) dt\ell(t_0,t_1)=\int_{t_0}^{t_1}\dot{\ell}(t)\,\mathrm{d}t for a log-likelihood-ratio density ℓ˙(t)\dot{\ell}(t) (nats per second). Let windows of length ww start at tj=jht_j=jh, j=0,1,…j=0,1,\ldots, every h=w/Jh=w/J seconds for an integer J≥1J\ge1 (J=1J=1: no overlap). Over a stretch of length T≫wT\gg w, the sum of the overlapping windows’ log-odds equals JJ times the log-odds of the stretch, up to edge terms of length <w<w. Hence (i) the sign, and so a fixed-length decision, is unchanged: overlap adds no information; (ii) the summed log-odds overstate the evidence by the factor JJ and must be multiplied by h/wh/w to stay calibrated; (iii) with that correction the sequential rule is the one of Prop. 8 checked every hh instead of every ww seconds, which only reduces the overshoot.

Proofs. Prop. 5. By Bayes’ rule and (b), the posterior ratio equals the likelihood ratio p(x1:n∣y=1,a1:n)/p(x1:n∣y=2,a1:n)p(x_{1:n}\mid y=1,a_{1:n})/p(x_{1:n}\mid y=2,a_{1:n}). By (a) each likelihood factorises over windows into ∏ip(xi∣y,ai)\prod_i p(x_i\mid y,a_i), so the logarithm of the ratio is ∑iℓi=Sn\sum_i\ell_i=S_n; hence P(y=1∣x1:n,a1:n)=Λ(Sn)P(y=1\mid x_{1:n},a_{1:n})=\Lambda(S_n). The minimum-error decision picks the larger posterior, i.e. sign⁡Sn\operatorname{sign}S_n. The majority vote depends on the signs of the ℓi\ell_i only; it agrees with sign⁡Sn\operatorname{sign}S_n for every configuration of signs only if all ∣ℓi∣|\ell_i| are equal.

Prop. 6. Let p1,p2p_1,p_2 be the densities of ℓi\ell_i under P1,P2P_1,P_2. Since ℓi\ell_i is the log-likelihood ratio of xix_i, the change of measure from P1P_1 to P2P_2 multiplies densities by e−ℓe^{-\ell}, so p2(ℓ)=e−ℓp1(ℓ)p_2(\ell)=e^{-\ell}p_1(\ell). With p1=N(μ,ν2)p_1=\mathcal{N}(\mu,\nu^2), p2p_2 integrates to one only if E1e−ℓi=e−μ+ν2/2=1\mathbb{E}_1e^{-\ell_i}=e^{-\mu+\nu^2/2}=1, i.e. ν2=2μ\nu^2=2\mu; completing the square then gives p2=N(μ−ν2,ν2)=N(−μ,2μ)p_2=\mathcal{N}(\mu-\nu^2,\nu^2)=\mathcal{N}(-\mu,2\mu), which is (i). (ii) holds by definition, DKL=E1log⁡p(xi∣y=1)p(xi∣y=2)=E1ℓi=μD_{\mathrm{KL}}=\mathbb{E}_1\log\frac{p(x_i\mid y=1)}{p(x_i\mid y=2)}=\mathbb{E}_1\ell_i=\mu. For (iii), independence gives Sn∣y=1∼N(nμ,2nμ)S_n\mid y=1\sim\mathcal{N}(n\mu,2n\mu), so P1(Sn>0)=Φ(nμ/2nμ)=Φ(nμ/2)P_1(S_n>0)=\Phi(n\mu/\sqrt{2n\mu})=\Phi(\sqrt{n\mu/2}); by (i) the same holds for P2(Sn<0)P_2(S_n<0). Substituting nμ=ITn\mu=IT and κ2=I/2\kappa^2=I/2 gives the claim.

Prop. 7. Summing: by Prop. 6(iii) the accuracy is Φ(nμ/2)\Phi(\sqrt{n\mu/2}), and μ/2=Φ−1(q)\sqrt{\mu/2}=\Phi^{-1}(q) by the definition of qq. Voting: the number of correctly signed windows is binomial with parameters nn and qq; by the central limit theorem the majority is right with probability Φ((nq−n/2)/nq(1−q))=Φ(n(2q−1)/(2q(1−q)))\Phi((nq-n/2)/\sqrt{nq(1-q)})=\Phi(\sqrt{n}(2q-1)/(2\sqrt{q(1-q)})). For small μ\mu, q−12≈φ(0)μ/2q-\frac{1}{2}\approx\varphi(0)\sqrt{\mu/2} with φ(0)=1/2π\varphi(0)=1/\sqrt{2\pi}, and q(1−q)≈14q(1-q)\approx\frac{1}{4}, so the argument of Φ\Phi is n2/πμ/2\sqrt{n}\sqrt{2/\pi}\sqrt{\mu/2} for voting against nμ/2\sqrt{n}\sqrt{\mu/2} for summing; equal accuracy needs nvote/nsum=π/2n_{\mathrm{vote}}/n_{\mathrm{sum}}=\pi/2 windows (the asymptotic relative efficiency of the sign test).

Prop. 8. NN is finite with probability one because SnS_n drifts to ±∞\pm\infty. (i) Changing measure as in the proof of Prop. 6, the probability of a wrong decision under y=2y=2 is P2(SN≥A)=E1[e−SN;SN≥A]P_2(S_N\ge A)=\mathbb{E}_1[e^{-S_N};S_N\ge A]. Without overshoot SN∈{−A,A}S_N\in\{-A,A\}, so this equals e−AP1(SN≥A)=e−A(1−α)e^{-A}P_1(S_N\ge A)=e^{-A}(1-\alpha), where α\alpha is the error rate; by the symmetry of (i) in Prop. 6 the error rate is the same under both hypotheses, so α=e−A(1−α)\alpha=e^{-A}(1-\alpha), i.e. α=1/(1+eA)\alpha=1/(1+e^A) and A=log⁡1−ααA=\log\frac{1-\alpha}{\alpha}. (ii) Wald’s identity gives E1SN=μE1N\mathbb{E}_1S_N=\mu\mathbb{E}_1N, and without overshoot E1SN=A(1−α)−Aα\mathbb{E}_1S_N=A(1-\alpha)-A\alpha, so E1N=A(1−2α)/μ\mathbb{E}_1N=A(1-2\alpha)/\mu windows (the same under P2P_2). Then E[TA]=wEN=A(1−2α)w/μ\mathbb{E}[T_A]=w\mathbb{E}N=A(1-2\alpha)w/\mu, and μ=Iw=2κ2w\mu=Iw=2\kappa^2w by Prop. 6 gives the stated time. (iii) is the Wald–Wolfowitz theorem for i.i.d. increments. TfixT_{\mathrm{fix}} solves Φ(κT)=1−α\Phi(\kappa\sqrt{T})=1-\alpha, and the ratio follows by division. As α→0\alpha\to0, log⁡1−αα∼log⁡1α\log\frac{1-\alpha}{\alpha}\sim\log\frac{1}{\alpha} and Φ−1(1−α)2∼2log⁡1α\Phi^{-1}(1-\alpha)^2\sim2\log\frac{1}{\alpha}, so the ratio tends to 1/41/4. With overshoot, ∣SN∣>A|S_N|>A: the realised error rate is below 1/(1+eA)1/(1+e^A) and the time above (ii), both by at most one window’s increment.

Prop. 9. Because NN is a stopping time, the event {N=n}\{N=n\} is determined by (x1:n,a1:n)(x_{1:n},a_{1:n}), so conditioning on it adds nothing to them: P(y=1∣x1:N,a1:N,N=n)=P(y=1∣x1:n,a1:n)=Λ(Sn)P(y=1\mid x_{1:N},a_{1:N},N=n)=P(y=1\mid x_{1:n},a_{1:n})=\Lambda(S_n) by Prop. 5. The decision sign SNS_N is therefore correct with probability Λ(∣SN∣)\Lambda(|S_N|), which is at least Λ(A)\Lambda(A) whenever ∣SN∣≥A|S_N|\ge A. This is the likelihood principle: the posterior does not depend on the stopping rule.

Prop. 10. Consider the stretch [0,T)[0,T) and let χ(t)\chi(t) be the number of windows [tj,tj+w)[t_j,t_j+w) inside it that contain the time tt. Every tt at distance at least ww from both ends of the stretch lies in exactly J=w/hJ=w/h windows, so χ(t)=J\chi(t)=J there, and χ(t)<J\chi(t)<J only within ww of the ends. Summing the window log-odds, ∑jℓ(tj,tj+w)=∫0Tℓ(t)χ(t) dt=Jℓ(0,T)+E\sum_j\ell(t_j,t_j+w)=\int_0^T\ell(t)\chi(t)\,\mathrm{d}t=J\ell(0,T)+E, where the edge term EE involves only data within ww of the ends. Hence the sum has the sign of the stretch’s log-likelihood ratio ℓ(0,T)\ell(0,T) up to EE (i), and it is JJ times the calibrated value (ii). Multiplied by 1/J=h/w1/J=h/w, the running sum after each new window equals the log-likelihood ratio of the data read so far, up to the edge term; this is the statistic SS of Prop. 8 evaluated every hh instead of every ww seconds (iii).

Related rules. Geirnaert et al. [21] turn fixed-window accuracies into the minimal expected switch duration of a gain-control Markov chain; the sequential rule can feed that chain the same way. Bayesian state-space filtering [33] additionally models attention switches, which our datasets do not contain. Calibration by one scale is temperature scaling [23]; averaging the calibrated log-odds of independently trained models is a deep ensemble [29].

Calibration and combining windows

Table 22. Raw margins are over-confident (expected calibration error, ECE, 0.08–0.17 on the two-talker sets); one scale fitted on validation brings every dataset to ECE ≤0.028\le0.028. Averaging the calibrated log-odds of the three seeds adds 1.6–2.1 points at 5 s. Summing a trial’s log-odds instead of majority voting changes accuracy by -0.1 to 4.2 points (Prop. 7). The calibrated 1-s log-odds have Var/(2 mean) = 0.99–1.13 (exactly 1 for Gaussian log-likelihood ratios) and lag-1 autocorrelation 0.05–0.05: consecutive windows are close to, but not exactly, independent.

DatasetECE (5 s) rawECE (5 s) cal.NLL, bits (5 s) rawNLL, bits (5 s) cal.acc. 5 s singleacc. 5 s ens.trial votetrial sum1-s log-odds var2 mean\frac{\mathrm{var}}{2\ \mathrm{mean}}1-s log-odds lag-1 rr
KU Leuven0.1130.0220.910.7872.774.397.699.31.110.05
DTU0.1110.0120.860.7076.278.394.697.01.040.05
Fuglsang’200.0780.0160.720.6479.681.296.898.50.990.05
ESAA0.0980.0160.820.6877.980.095.697.11.030.05
KUL-255ch0.1430.0281.040.8469.471.599.699.41.130.05
Broderick0.1710.0191.130.9063.665.778.983.11.030.05
SparrKULee0.0140.0020.280.2792.393.0––0.980.05
MEG-MASC0.0140.0010.050.0499.099.3––1.050.01

Table 22. Calibration and combination of windows (test folds; mean of 3 seeds unless marked). ECE: expected calibration error, 10 bins; NLL: log-loss in bits (chance 1). ens.: average of the 3 seeds’ calibrated log-odds. Trial: one decision per trial by majority vote of the 5-s windows or by the sign of their summed log-odds. Last two columns: diagnostics of Prop. 6 on the 1-s log-odds.

Sequential decisions

Figure 11 and Table 23 compare the sequential rule of Prop. 8 on calibrated 1-s log-odds with fixed-length decisions that sum the same 1-s log-odds over TT s, at equal mean decision time. Both rules see exactly the same scores; only the stopping differs. Thresholds AA are fixed values, not tuned. For CoupleFlow the sequential rule gains 5.1–5.6 points at A=1A=1 and 5.3–7.9 at A=2A=2; for the linear backward model it gains 3.8–5.5 and 3.3–8.0. The gain is therefore a property of the decision rule, available to any calibrated decoder; the ranking of decoders is set by κ\kappa, which is larger for CoupleFlow on every dataset except Broderick. The observed mean times are 0.92–1.08 times Wald’s prediction from the fitted κ\kappa (dotted lines). Overshoot on the 1-s grid lengthens the observed times; a varying information rate within trials shortens them, since the rule stops early on strong stretches, which the i.i.d. prediction ignores. “Time” counts speech windows; every decision also needs the EEG of the neural lag (up to 1.1 s) after its last window, for fixed and sequential rules alike. Decision times are spread: Table 23 gives their 90th percentile next to the mean.

Figure 11. Accuracy against mean decision time, two-talker datasets (test folds). Fixed: sign of the summed calibrated log-odds of TT consecutive 1-s windows. Sequential: the rule of Prop. 8 at thresholds A=0.25,…,5A=0.25,\ldots,5. Dotted: Wald’s prediction from CoupleFlow’s fitted κ\kappa (Prop. 8). CoupleFlow: mean of 3 seeds.

Datasetκ\kappaA=1A=1A=2A=2gainκ\kappaA=1A=1A=2A=2gain
KU Leuven0.24076.0 / 4.6 (9) / 70.586.6 / 10.9 (22) / 78.8+7.70.15870.8 / 6.7 (13) / 66.180.6 / 16.7 (34) / 74.0+6.6
DTU0.26777.0 / 4.1 (8) / 71.885.5 / 8.8 (17) / 78.8+6.70.21375.2 / 6.0 (12) / 70.283.3 / 13.3 (27) / 78.2+5.1
Fuglsang’200.31879.1 / 3.5 (7) / 73.587.5 / 7.3 (14) / 81.5+6.00.29979.2 / 4.0 (8) / 73.787.4 / 8.4 (16) / 82.0+5.4
ESAA0.29278.3 / 3.6 (7) / 73.287.3 / 7.7 (15) / 80.8+6.50.25077.0 / 4.6 (9) / 71.985.4 / 10.1 (20) / 80.0+5.4
KUL-255ch0.20374.9 / 5.9 (12) / 69.486.0 / 14.7 (30) / 78.1+7.90.15973.4 / 8.4 (17) / 68.384.3 / 21.5 (43) / 76.3+8.0
Broderick0.13172.1 / 10.4 (22) / 66.877.7 / 22.0 (50) / 72.4+5.30.14672.8 / 10.4 (21) / 69.078.6 / 22.2 (46) / 75.3+3.3

Table 23. Sequential against fixed-length decisions on calibrated 1-s log-odds. κ\kappa: fit of Φ(κT)\Phi(\kappa\sqrt{T}) to the fixed-length curve. Cells: sequential accuracy (%) / mean decision time (s) (90th percentile) / fixed-length accuracy at the same mean time (%). Gain: sequential minus fixed at A=2A=2 (points).

Anatomy of the sequential decisions

Table 24 and Figs. 12 and 13 open up the sequential rule on CoupleFlow’s calibrated 1-s log-odds (decisions of the three seeds pooled, test folds). Four things stand out.

The threshold sets the accuracy. Decisions that reach the threshold are about as accurate as the guarantee Λ(A)\Lambda(A) of Prop. 9 (Figure 12a; deviation -2.2 to +9.9 points over all datasets and thresholds). At small AA they are above it, because a 1-s step overshoots the threshold; at large AA slightly below it, the signature of the mild positive dependence between consecutive windows, which makes long sums a little over-confident. At A=1A=1 decisions are 74.9–80.2% correct (Λ(1)\Lambda(1) = 73.1%), at A=2A=2 86.1–90.6% (Λ(2)\Lambda(2) = 88.1%), on every dataset: a target accuracy fixes AA, to within a few points, without looking at the data.

Figure 12. Effect of the threshold AA. (a) Accuracy of the decisions that reached the threshold, against the guarantee Λ(A)\Lambda(A) (dashed). (b) Mean chosen length. (c) Share of decisions cut by the end of a trial.

The chosen length adapts widely. The same threshold yields short and long decisions (Figure 13a). At A=1A=1 the median decision uses 3–7 s and the 90th percentile 7–21 s; 3–20% of decisions are taken after a single second. At A=2A=2 the median is 6–17 s and the 90th percentile 14–49 s. The mean length grows with AA roughly as A(1−2α)2κ2\frac{A(1-2\alpha)}{2\kappa^2} (Prop. 8), so datasets with small κ\kappa (Broderick, KUL-255ch) wait longest (Figure 12b).

Figure 13. The chosen decision length. (a) Share of decisions taken within a given number of seconds, for four thresholds; markers: medians. (b) Accuracy of the decisions that reached the threshold, grouped by the length they used, for A=1,2,3A = 1, 2, 3. Dotted: Λ(A)\Lambda(A), the accuracy Prop. 9 guarantees for every length. Bins with fewer than 20 decisions are omitted.

Accuracy hardly depends on the chosen length. As Prop. 9 predicts, fast and slow decisions are about equally accurate (Figure 13b): decisions of at most 8 s are at most 1.9 points below Λ(2)\Lambda(2). The exception is the longest decisions (≥17\geq17 s), up to 9.7 points below Λ(2)\Lambda(2). They come from stretches in which the recording tracks the speech poorly, and one global calibration scale over-states the evidence of such windows: calibration holds on average, not conditionally on the local signal quality. A calibration that depends on the recent evidence rate would close this gap.

Finite trials force some decisions. When a trial ends before ∣S∣|S| reaches AA, the leftover is decided anyway (Figure 12c). These forced decisions are less accurate (58.4–79.6% at A∈{1,2}A \in\{1,2\}) and frequent on the short trials of Broderick (up to 37.6% at A=2A=2); the “all” column includes them. In continuous listening there is no trial end and they do not arise.

DatasetAAaccuracy (%) Λ(A)\Lambda(A)accuracy (%) allaccuracy (%) decidedchosen length (s) meanchosen length (s) medianchosen length (s) 10–90%forced at trial end share (%)forced at trial end acc. (%)vs. fixed length fixed (%)vs. fixed length gain
KU Leuven0.562.268.468.52.221–40.560.165.1+3.3
173.176.076.24.641–91.665.370.5+5.5
1.581.882.182.67.662–152.966.675.5+6.7
288.186.587.310.983–224.470.978.8+7.7
395.391.893.117.4135–357.274.183.9+7.9
498.294.396.423.7198–4710.075.987.3+7.0
DTU0.562.269.670.02.121–42.455.566.6+3.0
173.177.078.24.131–86.960.571.8+5.1
1.581.882.284.46.552–1312.166.475.7+6.6
288.185.588.98.873–1817.369.378.8+6.7
395.388.894.513.1114–2626.572.983.3+5.4
498.290.297.016.8146–3334.877.486.0+4.1
Fuglsang’200.562.271.872.11.911–42.054.668.3+3.5
173.179.180.23.531–75.459.773.5+5.6
1.581.884.286.35.442–119.363.878.2+6.0
288.187.590.67.362–1413.467.481.5+6.0
395.391.396.010.994–2120.973.485.9+5.4
498.292.498.013.9125–2627.377.788.6+3.9
ESAA0.562.270.971.11.911–41.355.867.6+3.3
173.178.379.13.631–73.858.773.2+5.1
1.581.883.885.15.642–116.865.077.8+6.0
288.187.389.57.762–159.866.680.8+6.5
395.391.094.211.594–2215.573.684.8+6.2
498.292.396.215.0125–2820.577.488.1+4.3
KUL-255ch0.562.266.866.82.621–50.461.263.5+3.3
173.174.874.95.942–121.266.769.3+5.5
1.581.881.281.410.083–212.271.874.5+6.7
288.185.986.114.6114–293.479.678.0+7.9
395.391.993.124.2188–495.873.583.5+8.4
498.294.796.133.12611–648.078.786.1+8.6
Broderick0.562.265.065.74.331–96.054.161.4+3.6
173.172.274.910.173–2116.558.466.9+5.3
1.581.875.981.416.1124–3427.261.170.1+5.8
288.177.986.921.6176–4937.662.872.5+5.4
395.379.793.530.6278–5653.667.775.0+4.7
498.279.996.637.13811–5665.471.175.0+4.9

Table 24. Anatomy of the sequential rule (CoupleFlow, calibrated 1-s log-odds, 3 seeds pooled, test folds). Λ(A)\Lambda(A): accuracy guaranteed by Prop. 9 for decisions that reach the threshold. Accuracy of all decisions and of those that reached the threshold (decided). Chosen length: mean, median and 10th–90th percentile of the number of seconds a decision used. Forced: share and accuracy of decisions cut by the end of a trial. Fixed: fixed-length accuracy at the same mean length; gain in points.

Overlapping windows

Table 25 and Fig. 14 repeat the analysis with 1-s windows every 0.5 s and 0.25 s (CoupleFlow, seed 0). The data follow Prop. 10 closely. The log-odds of windows jj apart correlate as $1-jh/w$ (KU Leuven, hop 0.25 s: 0.74, 0.48, 0.25 at j=1,2,3j=1,2,3; prediction 0.75, 0.5, 0.25), fixed-length accuracy changes by at most 0.7 points, and the summed log-odds are over-confident (5-s ECE up to 0.20) until scaled by h/wh/w (ECE $\le 0.02). The corrected sequential rule gains 0.1–1.6 points over the non-overlapping one at the same mean time, the small effect of checking the threshold more often. One detail matters: after a decision the rule must restart at the first window that shares no data with that decision. Restarting at the adjacent window re-uses the window that crossed the threshold and inflates accuracy at A=2A=2 by 0.5–2.7 points without any new evidence.

Training-window length

The paper’s models are trained on 5-s crops and applied at every test window. We retrained CoupleFlow with 1-, 2- and 10-s crops, and with a crop length redrawn from {1,2,5,10}\{1, 2, 5, 10\} s at every epoch (“mixed”), keeping everything else fixed: the same folds, seeds, losses, epochs, and model selection on 5-s validation windows (Figure 15, Table 26).

Figure 15. Test accuracy against test window for CoupleFlow trained on 1-, 2-, 5- (paper), 10-s crops or a length redrawn from {1,2,5,10}\{1, 2, 5, 10\} s every epoch; everything else as in the paper (selection on 5-s validation windows). Mean of 3 seeds, test folds; competing talker for two-talker sets, match–mismatch for SparrKULee and MEG-MASC.

Datasettraining window1 s5 s30 sκ\kappa (1-s sums)seq. A=2A=2: acc. / time
KU Leuven1 s62.1±0.362.1\pm0.372.9±0.672.9\pm0.689.4±0.289.4\pm0.20.25185.4 / 8.1
2 s61.5±0.261.5\pm0.273.4±0.673.4\pm0.691.2±0.391.2\pm0.30.25286.2 / 9.3
5 s (paper)60.5±0.160.5\pm0.172.7±0.272.7\pm0.290.5±0.490.5\pm0.40.24086.6 / 10.9
10 s60.2±0.160.2\pm0.172.2±0.672.2\pm0.690.3±0.590.3\pm0.50.23186.4 / 11.8
mixed 1–10 s62.0±0.362.0\pm0.374.0±0.474.0\pm0.491.5±0.691.5\pm0.60.25986.5 / 8.7
DTU1 s63.9±0.063.9\pm0.077.1±0.377.1\pm0.394.3±0.294.3\pm0.20.30886.9 / 6.6
2 s62.5±0.162.5\pm0.176.9±0.576.9\pm0.594.8±0.894.8\pm0.80.28786.2 / 7.7
5 s (paper)61.4±0.361.4\pm0.376.2±0.676.2\pm0.694.5±0.894.5\pm0.80.26785.5 / 8.8
10 s61.1±0.261.1\pm0.275.3±0.375.3\pm0.393.7±0.893.7\pm0.80.25985.3 / 9.8
mixed 1–10 s63.2±0.163.2\pm0.178.1±0.278.1\pm0.295.4±0.295.4\pm0.20.30486.5 / 6.9
Fuglsang’201 s65.4±0.165.4\pm0.178.8±0.278.8\pm0.295.1±0.595.1\pm0.50.33187.5 / 6.0
2 s64.5±0.364.5\pm0.379.9±0.479.9\pm0.496.5±0.296.5\pm0.20.33087.5 / 6.6
5 s (paper)63.8±0.363.8\pm0.379.6±0.879.6\pm0.895.9±1.395.9\pm1.30.31887.5 / 7.3
10 s63.4±0.263.4\pm0.278.5±0.178.5\pm0.195.4±0.395.4\pm0.30.31287.7 / 7.8
mixed 1–10 s64.9±0.264.9\pm0.279.9±0.479.9\pm0.496.2±0.296.2\pm0.20.33987.9 / 6.4
ESAA1 s65.1±0.165.1\pm0.178.3±0.478.3\pm0.494.2±0.694.2\pm0.60.31687.2 / 6.0
2 s64.0±0.364.0\pm0.378.4±0.478.4\pm0.493.9±0.393.9\pm0.30.30987.6 / 6.9
5 s (paper)63.2±0.363.2\pm0.377.9±0.377.9\pm0.393.4±0.593.4\pm0.50.29287.3 / 7.7
10 s62.3±0.462.3\pm0.476.4±0.176.4\pm0.192.4±0.392.4\pm0.30.27587.0 / 9.0
mixed 1–10 s64.7±0.164.7\pm0.179.2±0.179.2\pm0.194.2±0.494.2\pm0.40.31087.4 / 6.4
KUL-255ch1 s59.6±0.359.6\pm0.368.3±0.268.3\pm0.285.9±0.485.9\pm0.40.20084.2 / 12.3
2 s59.0±0.359.0\pm0.369.3±0.669.3\pm0.686.4±0.686.4\pm0.60.20884.7 / 13.2
5 s (paper)58.9±0.358.9\pm0.369.4±0.469.4\pm0.487.7±0.987.7\pm0.90.20386.0 / 14.7
10 s58.6±0.358.6\pm0.369.4±0.469.4\pm0.488.1±1.288.1\pm1.20.20185.1 / 14.4
mixed 1–10 s59.3±0.259.3\pm0.270.1±0.370.1\pm0.387.7±0.687.7\pm0.60.21485.5 / 12.7
Broderick1 s56.8±0.356.8\pm0.364.0±0.564.0\pm0.578.8±0.378.8\pm0.30.14979.4 / 17.6
2 s56.6±0.556.6\pm0.565.4±1.065.4\pm1.080.1±2.680.1\pm2.60.14680.0 / 18.9
5 s (paper)55.8±0.755.8\pm0.763.6±1.863.6\pm1.879.2±2.679.2\pm2.60.13177.7 / 22.0
10 s55.3±0.955.3\pm0.962.6±2.162.6\pm2.176.7±3.576.7\pm3.50.11976.6 / 24.1
mixed 1–10 s57.2±0.457.2\pm0.466.2±0.666.2\pm0.683.0±1.083.0\pm1.00.16081.0 / 16.7
SparrKULee1 s75.4±0.175.4\pm0.190.9±0.190.9\pm0.199.5±0.099.5\pm0.0––
2 s74.6±0.174.6\pm0.192.3±0.192.3\pm0.199.7±0.099.7\pm0.0––
5 s (paper)73.5±0.073.5\pm0.092.2±0.092.2\pm0.099.7±0.199.7\pm0.1––
10 s71.9±0.071.9\pm0.091.5±0.191.5\pm0.199.7±0.099.7\pm0.0––
mixed 1–10 s74.4±0.274.4\pm0.292.2±0.092.2\pm0.099.7±0.099.7\pm0.0––
MEG-MASC1 s89.5±0.289.5\pm0.298.9±0.198.9\pm0.1100.0±0.0100.0\pm0.0––
2 s87.8±0.387.8\pm0.399.3±0.199.3\pm0.1100.0±0.0100.0\pm0.0––
5 s (paper)83.7±0.483.7\pm0.499.0±0.199.0\pm0.1100.0±0.0100.0\pm0.0––
10 s79.4±0.579.4\pm0.597.9±0.197.9\pm0.1100.0±0.0100.0\pm0.0––
mixed 1–10 s87.5±0.787.5\pm0.799.2±0.199.2\pm0.1100.0±0.0100.0\pm0.0––

Table 26. Training-window ablation: accuracy (%, mean±\pms.d. over 3 seeds) at 1-, 5- and 30-s test windows, κ\kappa of the summed 1-s log-odds, and the sequential rule at A=2A=2 (two-talker sets).

Figure 14. Overlapping 1-s windows (CoupleFlow, seed 0, test folds): the sequential rule on h/wh/w-scaled log-odds with hop h=1h = 1, 0.5 and 0.25 s, restarting after each decision at the first window disjoint from it, and fixed-length decisions (nearly identical for every hop; the hop-1 s curve is shown).

Datasethop 1 s fixed 5 shop 1 s ECE raw/corr.hop 1 s seq. A=2A=2hop 0.5 s fixed 5 shop 0.5 s ECE raw/corr.hop 0.5 s seq. A=2A=2hop 0.25 s fixed 5 shop 0.25 s ECE raw/corr.hop 0.25 s seq. A=2A=2
KU Leuven71.50.04/0.0485.9 / 10.471.80.12/0.0287.1 / 11.0 (+0.5)71.60.19/0.0287.1 / 11.1 (+0.4)
DTU73.70.02/0.0285.5 / 8.774.20.10/0.0186.9 / 9.4 (+0.6)74.30.17/0.0187.2 / 9.7 (+0.6)
Fuglsang’2077.90.01/0.0187.5 / 7.277.30.10/0.0188.2 / 7.7 (+0.2)77.20.15/0.0188.6 / 7.9 (+0.3)
ESAA76.60.02/0.0287.3 / 7.876.60.10/0.0188.0 / 8.3 (+0.1)76.60.16/0.0188.7 / 8.5 (+0.4)
KUL-255ch67.50.03/0.0386.3 / 16.068.00.11/0.0288.5 / 17.2 (+1.2)67.60.20/0.0188.7 / 17.3 (+1.3)
Broderick61.00.03/0.0375.9 / 24.061.10.10/0.0277.2 / 25.7 (+0.3)60.80.19/0.0278.7 / 26.2 (+1.6)

Table 25. Overlapping 1-s windows (CoupleFlow, seed 0). Fixed 5 s: accuracy of the summed log-odds of the windows inside each 5-s stretch. ECE of those sums without (raw) and with the h/wh/w correction. Seq.: corrected sequential rule at A=2A=2, accuracy (%) / mean time (s).

Differences below are against the paper’s 5-s training, mean of 3 seeds, over all 8 datasets.

Short crops help short windows. Training on 1-s crops raises 1-s accuracy by +0.8 to +5.9 points (n=8n = 8 datasets), but changes 5-s accuracy by -1.3 to +0.9 and 30-s accuracy by -1.8 to +0.8. Long crops help nothing: 10-s training changes 1-s accuracy by -4.3 to -0.2 and 5-s accuracy by -1.4 to -0.1 points.

Mixed crops are best overall. Redrawing the crop length raises 1-s accuracy on every dataset (+0.4 to +3.8 points, n=8n = 8) and lowers neither 5-s accuracy (0.0 to +2.6) nor 30-s accuracy (0.0 to +3.8) by more than 0.1 points. In the terms of Prop. 6 it raises the information rate of the 1-s evidence: κ\kappa of the summed 1-s log-odds grows by a factor 1.05–1.22, and the sequential rule at A=2A=2 needs 0.76–0.88 of the time (0.74–0.84 for 1-s training). A plausible reading, which we have not tested further: a model that only sees long crops learns a coupling that is reliable when averaged over many seconds but noisy within one; short crops force every second to carry evidence, and mixing lengths keeps the long-window accuracy. The paper’s main results use 5-s training throughout; mixed-length training is a free improvement for any window.

References

  1. [1]Accou, B., Bollens, L., Gillis, M., Verheijen, W., Van hamme, H., and Francart, T. (2024). Spar-rKULee: A speech-evoked auditory response repository from KU Leuven, containing the EEG of 85 participants. Data, 9(8):94.
  2. [2]Accou, B., Jalilpour Monesi, M., Montoya, J., Van hamme, H., and Francart, T. (2020). Modeling the relationship between acoustic stimulus and EEG with a dilated convolutional neural network. In 28th European Signal Processing Conference (EUSIPCO), pages 1175–1179.DOI
  3. [3]Accou, B., Vanthornhout, J., Van hamme, H., and Francart, T. (2023). Decoding of the speech envelope from EEG using the VLAAI deep neural network. Scientific Reports, 13:812.DOI
  4. [4]Anonymous (2026). MERIT: Making every modality earn its contribution in multimodal attention decoding. Concurrent submission under review.
  5. [5]Bardes, A., Ponce, J., and LeCun, Y. (2022). VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations.arxiv.org/abs/2105.04906
  6. [6]Biesmans, W., Das, N., Francart, T., and Bertrand, A. (2017). Auditory-inspired speech envelope extraction methods for improved EEG-based auditory attention detection in a cocktail party scenario. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(5):402–412.DOI
  7. [7]Bogacz, R., Brown, E., Moehlis, J., Holmes, P., and Cohen, J. D. (2006). The physics of optimal decision making: A formal analysis of models of performance in two-alternative forced-choice tasks. Psychological Review, 113(4):700–765.pubmed.ncbi.nlm.nih.gov/17014301
  8. [8]Brodbeck, C., Jiao, A., Hong, L. E., and Simon, J. Z. (2020). Neural speech restoration at the cocktail party: Auditory cortex recovers masked speech of both attended and ignored speakers. PLOS Biology, 18(10).DOI
  9. [9]Broderick, M. P., Anderson, A. J., Di Liberto, G. M., Crosse, M. J., and Lalor, E. C. (2018). Electrophysiological correlates of semantic dissimilarity reflect the comprehension of natural, narrative speech. Current Biology, 28(5):803–809.DOI
  10. [10]Crosse, M. J., Di Liberto, G. M., Bednar, A., and Lalor, E. C. (2016). The multivariate temporal response function (mTRF) toolbox: A MATLAB toolbox for relating neural signals to continuous stimuli. Frontiers in Human Neuroscience, 10.pubmed.ncbi.nlm.nih.gov/27965557
  11. [11]Das, N., Biesmans, W., Bertrand, A., and Francart, T. (2016). The effect of head-related filtering and ear-specific decoding bias on auditory attention detection. Journal of Neural Engineering, 13(5).DOI
  12. [12]de Cheveigné, A., Wong, D. D. E., Di Liberto, G. M., Hjortkjær, J., Slaney, M., and Lalor, E. (2018). Decoding the auditory brain with canonical component analysis. NeuroImage, 172:206–216.DOI
  13. [13]Decruy, L., Vanthornhout, J., and Francart, T. (2020). Hearing impairment is associated with enhanced neural tracking of the speech envelope. Hearing Research, 393:107961.DOI
  14. [14]Défossez, A., Caucheteux, C., Rapin, J., Kabeli, O., and King, J.-R. (2023). Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5:1097–1107.DOI
  15. [15]Ding, N. and Simon, J. Z. (2012). Emergence of neural encoding of auditory objects while listening to competing speakers. Proceedings of the National Academy of Sciences, 109(29):11854–11859.DOI
  16. [16]Fan, C., Yang, X., Zhang, H., Chen, Y., Li, L., Zhou, J., and Lv, Z. (2025). ListenNet: A lightweight spatio-temporal enhancement nested network for auditory attention detection. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI).arxiv.org/abs/2505.10348
  17. [17]Fiedler, L., Wöstmann, M., Graversen, C., Brandmeyer, A., Lunner, T., and Obleser, J. (2017). Single-channel in-ear-EEG detects the focus of auditory attention to concurrent tone streams and mixed speech. Journal of Neural Engineering, 14(3):036020.DOI
  18. [18]Fuglsang, S. A., Dau, T., and Hjortkjær, J. (2017). Noise-robust cortical tracking of attended speech in real-world acoustic scenes. NeuroImage, 156:435–444.DOI
  19. [19]Fuglsang, S. A., Märcher-Rørsted, J., Dau, T., and Hjortkjær, J. (2020). Effects of sensorineural hearing loss on cortical synchronization to competing speech during selective attention. Journal of Neuroscience, 40(12):2562–2572.DOI
  20. [20]Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. (2016). Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35.arxiv.org/abs/1505.07818
  21. [21]Geirnaert, S., Francart, T., and Bertrand, A. (2020). An interpretable performance metric for auditory attention decoding algorithms in a context of neuro-steered gain control. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(1):307–317.DOI
  22. [22]Geirnaert, S., Vandecappelle, S., Aliciovic, E., de Cheveigné, A., Lalor, E., Meyer, B. T., Miran, S., Francart, T., and Bertrand, A. (2021). Electroencephalography-based auditory attention decoding: Toward neurosteered hearing devices. IEEE Signal Processing Magazine, 38(4):89–102.DOI
  23. [23]Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1321–1330.arxiv.org/abs/1706.04599
  24. [24]Gwilliams, L., Flick, G., Marantz, A., Pylkkänen, L., Poeppel, D., and King, J.-R. (2023). Introducing MEG-MASC, a high-quality magnetoencephalography dataset for evaluating natural speech processing. Scientific Data, 10:862.
  25. [25]Hassan, K. M. N., Alavi, A., and Williamson, D. S. (2026). MAESTRO: a multimodal auditory-attention egocentric speech-TRacking open corpus. arXiv preprint arXiv:2609.31898.arxiv.org/abs/2609.31898
  26. [26]Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J.-D., Blankertz, B., and Bießmann, F. (2014). On the interpretation of weight vectors of linear models in multivariate neuroimaging. NeuroImage, 87:96–110.DOI
  27. [28]Kapoor, S. and Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9).DOI
  28. [29]Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30.arxiv.org/abs/1612.01474
  29. [31]Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. (2023). Flow matching for generative modeling. In International Conference on Learning Representations.arxiv.org/abs/2210.02747
  30. [32]Mesgarani, N. and Chang, E. F. (2012). Selective cortical representation of attended speaker in multi-talker speech perception. Nature, 485(7397):233–236.DOI
  31. [33]Miran, S., Akram, S., Sheikhattar, A., Simon, J. Z., Zhang, T., and Babadi, B. (2018). Real-time tracking of selective auditory attention from M/EEG: A Bayesian filtering approach. Frontiers in Neuroscience, 12:262.DOI
  32. [34]Mirkovic, B., Debener, S., Jaeger, M., and De Vos, M. (2015). Decoding the attended speech stream with multi-channel EEG: implications for online, daily-life applications. Journal of Neural Engineering, 12(4):046007.DOI
  33. [35]Mundanad Narayanan, A., Zink, R., and Bertrand, A. (2021). EEG miniaturization limits for stimulus decoding with EEG sensor networks. Journal of Neural Engineering, 18.DOI
  34. [37]O’Sullivan, J. A., Power, A. J., Mesgarani, N., Rajaram, S., Foxe, J. J., Shinn-Cunningham, B. G., Slaney, M., Shamma, S. A., and Lalor, E. C. (2015). Attentional selection in a cocktail party environment can be decoded from single-trial EEG. Cerebral Cortex, 25(7):1697–1706.
  35. [38]Petersen, E. B., Wöstmann, M., Obleser, J., and Lunner, T. (2017). Neural tracking of attended versus ignored speech is differentially affected by hearing loss. Journal of Neurophysiology, 117(1):18–27.
  36. [39]Puffay, C., Accou, B., Bollens, L., Jalilpour Monesi, M., Vanthornhout, J., Van hamme, H., and Francart, T. (2023). Relating EEG to continuous speech using deep neural networks: a review. Journal of Neural Engineering, 20(4).arxiv.org/abs/2302.01736
  37. [40]Rotaru, I., Geirnaert, S., Heintz, N., Van de Ryck, I., Bertrand, A., and Francart, T. (2024). What are we really decoding? Unveiling biases in EEG-based decoding of the spatial focus of auditory attention. Journal of Neural Engineering, 21(1).
  38. [41]Vandecappelle, S., Deckers, L., Das, N., Ansari, A. H., Bertrand, A., and Francart, T. (2021). EEG-based detection of the locus of auditory attention with convolutional neural networks. eLife, 10:e56481.
  39. [42]Wald, A. (1945). Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics, 16(2):117–186.
  40. [43]Wald, A. and Wolfowitz, J. (1948). Optimum character of the sequential probability ratio test. The Annals of Mathematical Statistics, 19(3):326–339.DOI
  41. [44]Yan, S., Fan, C., Zhang, H., Yang, X., Tao, J., and Lv, Z. (2024). DARNET: Dual attention refinement network with spatiotemporal construction for auditory attention detection. In Advances in Neural Information Processing Systems.arxiv.org/abs/2410.11181
  42. [45]Zhang, Y., Ruan, H., Yuan, Z., Du, H., Gao, X., and Lu, J. (2023). A learnable spatial mapping for decoding the directional focus of auditory attention using EEG. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).DOI

Paper details

vlnc:W_gtdnb28q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Claude

View statement
Looking for related papers…
Contents