WorksComputer Science
Who Called? V33DA: A Physically Verified Multimodal Benchmark for Vocal Attribution in Zebra Finch Groups
Abstract
Deciding which member of an animal group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3D candidate geometry. We close this gap with V33DA, a dataset-centered multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an on-body reference ceiling. Given a detected vocalization and the visible candidate birds, the task is to select the caller. We evaluate it under three regimes: session-disjoint testing on familiar birds from held-out recording dates, held-out-experiment transfer to a new cohort, and leave-one-individual-out evaluation on a bird whose vocalizations are held out from training. Reference methods expose a consistent gap between in-domain accuracy and transfer to unseen individuals: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The on-body reference stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels. We release V33DA with fixed protocols and code for the evaluated methods, and V33DA++, an unevaluated companion set of verified overlapping-call events and +/-2 s context windows.
Introduction
Vocal communication in a social group is only interpretable at the level of individuals once each call can be assigned to the individual that produced it. In zebra finches, vocalizations carry information about identity, pair bonds, and social context [14, 15]; without caller attribution, these signals cannot be linked back to the animals that generated them. The problem is therefore the entry point to studying who communicates with whom, when, and under what social conditions.
Caller attribution in freely behaving groups is difficult because the evidence is distributed across modalities. The sound arrives at several microphones, the candidate animals move through a shared 3D space, and visual identity, posture, proximity, and acoustic structure all change over time. A model can also solve the wrong problem: if the same individuals appear in training and test, high accuracy may reflect recognition of familiar voices rather than a transferable ability to assign a call to the correct visible animal. A useful benchmark must therefore separate familiar-individual recognition from spatial attribution that generalizes to new individuals.
Existing resources capture important pieces of this problem but not the full setting. Large bioacoustic corpora support species, call-type, or individual-recognition tasks, but typically do not provide the synchronized candidate geometry needed to decide which animal in a group produced a sound [31, 26, 49]. Spatial-audio benchmarks evaluate localization or direction-of-arrival estimation, but usually do not provide physically verified caller identity for freely behaving animals [16, 46, 56]. Active-speaker and diarization datasets ask related attribution questions in humans, but rely on perceptual annotation, face or mouth cues, or close-talk references, and rarely provide calibrated 3D candidate state [51, 41] (Xu et al. 2022). On-body animal-recording studies can verify who called, but they do not by themselves define a public multimodal benchmark for testing transfer across unseen individuals. What is missing is a dataset that combines physical verification, synchronized group state, and evaluation splits designed to expose identity shortcuts.
We introduce V33DA (Vocal 3-Experiment 3D Attribution), a dataset-centered multimodal benchmark for spatial vocal attribution in social zebra finches, built on the BirdPark recording system [52]. Each of the 33,625 released events includes 5-channel audio, three calibrated camera views, per-frame 3D pose for every visible candidate, per-bird radio telemetry, and an on-body accelerometer-derived vibration channel used for verification. The verification channel is withheld from benchmark models and used only to define an on-body reference ceiling. We additionally provide V33DA++ as an unevaluated companion release: 3,784 verified overlapping-call events excluded from the main V33DA benchmark and ±2 s padded context windows. These files are intended to support future work on source separation, vocal activity detection, and longer-context active-speaker tasks; no result in this paper depends on them.
The primary benchmark task is candidate-conditioned caller attribution: given a detected vocalization and the currently visible candidate birds, choose which bird produced the call. We also report 3D source localization as a diagnostic of whether a method has learned usable spatial evidence rather than only a candidate ranking. Evaluation uses three regimes: session-disjoint testing on familiar birds from held-out recording dates, leave-one-bird-out testing on a bird whose vocalizations are held out under the same cage and sensor setup, and held-out-experiment testing on a new cohort with no birds shared between train and test. Together, these regimes distinguish in-domain reuse, unseen-vocalizer generalization under fixed acquisition conditions, and broader experiment-level transfer.
The paper makes three contributions:
A physically verified multimodal release. V33DA provides synchronized audio, video, 3D pose, radio telemetry, and withheld on-body verification for social vocal attribution.
Splits that separate familiar from unseen individuals. The evaluation protocol separates familiar-individual testing from unseen-individual and held-out-experiment transfer, with fixed splits and common attribution/localization metrics.
A benchmark lesson about shortcut learning. Reference methods show that high in-domain accuracy can come from familiar-identity cues, while transfer to unseen individuals depends much more strongly on candidate-aware spatial reasoning.
We release the dataset, evaluation code, and implementations of the evaluated methods.
Related Work
Task landscape. Several tasks touch the problem of assigning vocalizations to individuals, but they differ in what is given and what must be predicted. Individual vocal identification (ID) [15, 45, 39] starts from an isolated vocalization and predicts the identity of a known caller; the scene and the other possible callers are usually absent. Sound-source localization (SSL) [32, 16, 58] starts from array audio and predicts a position or direction of arrival; it can recover where the sound came from without deciding which animal at that position called. Source separation (Sep) and vocal activity detection (VAD) [53, 54] operate over longer audio: they ask which sources are active, when they are active, and sometimes how to separate their waveforms, but they do not necessarily attach each event to a tracked individual. Active-speaker detection (ASD) and audio-visual diarization (AVD) [51, 8] (Xu et al. 2022) ask which visible person is speaking over time, most often in human video where faces, mouths, or close-talk microphones provide supervision.
V33DA targets candidate-conditioned caller attribution (CA). The event has already been detected, the candidate animals visible at that moment are known, and the model must choose which candidate produced the call. This is different from ID because the test caller may be unfamiliar, different from SSL because the output must be an individual rather than only a point in space, and different from tracking because the candidates are already tracked. The task therefore needs both verified caller identity and synchronized scene geometry: recognizing a familiar voice is not enough under unseen-individual transfer, and localizing a sound is not enough unless the location can be linked to a candidate animal. Appendix Table 5 maps these tasks onto existing resources.
| Audio tasks | Multimodal | Video tasks | Radio | ||||||||
| SSL | Sep | VAD | ID | CA | ASD | KP | Pose | Track | Rad | Transfer | |
| Anisimov+, Gill+ [2014, 2016] | – | ✓ | – | – | ✓ | – | – | – | – | – | – |
| Rookognise [2022] | – | – | ✓ | ✓ | ✓ | – | – | – | – | – | – |
| MeerKAT [2025] | – | – | ✓ | – | – | – | – | – | – | – | – |
| Prat+ [2017] | – | – | – | – | ✓ | – | – | – | – | – | – |
| HyVL [2023] | ✓ | ✓ | ✓ | – | ✓ | – | – | – | ✓ | – | – |
| VCL [2024] | ✓ | ✓ | ✓ | – | ✓ | – | – | – | – | – | – |
| Elie+, Phaniraj+ [2018, 2023] | – | – | – | ✓ | – | – | – | – | – | – | – |
| VoxCeleb [2020] | – | – | – | ✓ | – | – | – | – | – | – | ✓ |
| AVA-ActSp., UniTalk [2019, 2025] | – | – | – | – | – | ✓ | – | – | – | – | ✓ |
| AVA-AVD [2022] | – | – | – | – | ✓ | ✓ | – | – | – | – | ✓ |
| Ego4D [2022] | – | – | – | – | – | ✓ | – | – | – | – | ✓ |
| AMI [2006] | – | ✓ | ✓ | ✓ | ✓ | ✓ | – | – | – | – | – |
| DIHARD II [2019] | – | – | ✓ | – | – | – | – | – | – | – | ✓ |
| VoxConverse [2020] | – | – | ✓ | – | ✓ | ✓ | – | – | – | – | ✓ |
| STARSS23 [2023] | ✓ | – | ✓ | – | – | – | – | – | – | – | – |
| DeepLabCut, SLEAP, idTracker [2018, 2022, 2019] | – | – | – | – | – | – | ✓ | – | ✓ | – | – |
| 3D-POP, 3D-MuPPET [2023, 2024] | – | – | – | – | – | – | ✓ | ✓ | ✓ | – | – |
| CalMS21 [2021] | – | – | – | – | – | – | ✓ | – | ✓ | – | – |
| V33DA (ours) | ✓⋆ | ✓† | ✓† | ✓ | ✓⋆ | ✓† | ✓ | ✓ | ✓† | ✓ | ✓⋆ |
Table 5. Task coverage of V33DA versus related resources. A checkmark indicates that the released data directly support the task. In the V33DA row, a star (⋆) marks tasks evaluated in this paper, and a dagger (†) marks tasks for which the unevaluated V33DA++ companion release provides data.
Dataset families. Existing resources cover different parts of this task landscape. Bioacoustic recognition datasets, models, and benchmarks provide species, call-type, or individual labels at scale, but generally lack synchronized group geometry: they can tell whether a vocal signature identifies an individual, not which visible animal in a shared scene produced a particular event [31, 26, 49, 15, 45, 7, 3]. Spatial-audio and sound event localization and detection (SELD) datasets provide localization or direction-of-arrival targets, including real and simulated array recordings, but usually do not provide physically verified animal caller identity [46, 16, 23, 56]. Human active-speaker and diarization datasets ask related attribution questions, but rely on human scenes, perceptual labels, face or mouth cues, or close-talk references rather than calibrated 3D state for all candidate animals [51, 41, 6, 21, 53, 8] (Xu et al. 2022). On-body animal-recording studies physically verify callers, but they are typically not public multimodal benchmarks combining room audio, synchronized video, 3D pose, radio telemetry, and transfer splits [2, 20, 54, 37, 47].
Multi-animal vision datasets provide the candidate state that attribution needs but not the vocal evidence itself. DeepLabCut, SLEAP, and idTracker anchor 2D pose and identity tracking [38, 43, 50]; 3D-POP and 3D-MuPPET extend to multi-view 3D skeletons [40] (Waldmann et al. 2024); and PAIR-R24M, MammalPaps, and CalMS21 scale up pairwise or social-behavior annotation [36, 17] (Sun et al. 2021). These resources are essential for reconstructing who is where, but caller attribution asks a different question: which of those candidates produced the audio event?
Radio and transfer. Multi-antenna radio for spatial reasoning is well-established outside bioacoustics: cellular and 5G exploit angle-of-arrival, time-difference-of-arrival, and channel-state information for meter- to centimeter-scale positioning [del Peral-Rosado et al., 2018, Wymersch et al., 2017], and WiFi/mmWave channel-state sensing carries the same principles to indoor localization, gesture recognition, and presence detection [34]. To our knowledge no prior public animal-vocalization dataset ships per-individual radio telemetry alongside calibrated position and audio-visual channels. Separating in-domain generalization from distribution shift is standard in transfer benchmarks such as DomainBed and WILDS [25, 33]; V33DA brings the same logic to social vocal attribution by separating session, leave-one-bird-out, and held-out-experiment regimes.
V33DA’s position. What remains missing is a public benchmark that brings these ingredients into the same event: synchronized state for every visible candidate, physically verified caller labels, and evaluation splits that test whether attribution transfers beyond familiar individuals. V33DA combines 5-channel audio, multi-view video, per-candidate 3D pose, per-bird radio telemetry, and an on-body verification channel that is withheld from benchmark models. Related resources support parts of this protocol but not the same verified evaluation: STARSS23 has no tracked animal candidates or caller identities, Rookognise has no synchronized trajectories or calibrated 3D geometry, and VCL has no independently verified dyadic caller labels. HyVL comes closest, but it assigns callers by matching acoustic localization to tracked snouts, so localization-first methods evaluated on it would be scored against localization-derived labels; the withheld on-body channel avoids this circularity. With 33,625 verified vocalizations, V33DA is also larger than Rookognise (about 17.7k) and HyVL (about 13.7k). Table 1 and Appendix Table 5 summarize how this combination differs from prior resources.
| Domain | Task | Size | N | Conc. | Mics | Vis. | Verif. | |
| Physically verified animal caller attribution and localization | ||||||||
| Anisimov+ [2014] | Zebra finch | CA | methods demo | 9 | 9 | 1/bird | – | on-body accel. |
| Gill+ [2016] | Zebra finch | CA | 182k calls | 8 | 8 | 1/bird | – | body-mounted mic |
| Roachson [2022] | Rook | CA + ID | 17.7k vocs | 15 | 15 | 6 | – | obs. + MC audio |
| MeerKAT [2025] | Meerkat | SSL + classif. | 251k events | 41 | 1/collar | body-mounted mic | ||
| Prat+ [2017] | Fruit bat | repertoire | 90k events | 33 | 2–8 | 1/chamb. | ✓ | mouth or IR video |
| HyVL [2023] | Mouse | SSL | 13.7k USVs | 18 | 1–3 | 68 | ✓ | beamforming + pose |
| VCL [2024] | Rodent | SSL | 767k events | – | 1–2 | 4–24 | ✓ | video pose / earbud |
| Individual vocal recognition | ||||||||
| Elie+ [2018] | Zebra finch | ID | 8k+ calls | 45 | 1 | 1 | – | isolated rec. |
| Phaniraj+ [2023] | Marmoset | ID | 7.1k calls | 20 | 1 | 1 | – | isolated rec. |
| VoxCeleb [2020] | Human | ID + verif. | 1M+ utt. | 7363 | 1 | 1 | ○ | SyncNet + face recog. |
| Human active-speaker, diarization, and spatial audio | ||||||||
| AVA-ActSP. [2019] | Human | ASD | 29.6k turns | – | 1–3 | 1 | ○ | perceptual |
| AVA-AVD [2022] | Human | AVD | 40.3k segs | 1500 | 2–24 | 1 | ○ | perceptual |
| UniTalk [2025] | Human | ASD | 44.5 h | 48693 | 2–3 | 1 | ○ | perceptual |
| AMI [2006] | Human | ASR + diar. | 93.6k segs | – | 4 | 8 + close-talk | ✓ | close-talk |
| DIHARD II [2019] | Human | diar. | 386 recs | – | 3–7 | 1 | – | perceptual |
| VoxConverse [2020] | Human | AVD | 27.7k turns | – | 2+ | 1 | ○ | SyncNet + perceptual |
| STARSS23 [2023] | General | SELD | 17.6k events | 57 | varies | 4 (FOA/MIC) | ✓ | mocap |
| V33DA (ours) | Zebra finch | CA + SSL + ID | 33.6k events | 10 | 3–4 | 5 | ✓ | on-body accel. |
Table 1. Comparison of V33DA with related datasets. Task abbreviations are defined in the text (classif. = classification, diar. = diarization, verif. = speaker verification). N: number of individuals; Conc.: individuals recorded concurrently; Mics: microphones; Vis.: synchronized video (○: in-the-wild video without calibrated geometry); Verif.: source of the caller label (on-body accel. = accelerometer on the animal; body-mounted mic = microphone on the animal; perceptual = human annotation); vocs = vocalizations, utt. = utterances, segs = segments.
Dataset
Recording setting. All recordings come from the BirdPark aviary [52], a synchronized multimodal installation for observing small social groups under naturalistic interaction. The benchmark spans three experiments sharing the same sensing configuration, each housing 3–4 zebra finches. Five calibrated microphones at known 3D positions record at 24,414 Hz, three synchronized cameras (top, back, side) capture video at 47.68 fps, and 3D pose is triangulated from 5 keypoints per bird. A 4-antenna receiver array provides per-bird radio telemetry, and an FM-demodulated on-body signal exposes thoracic vibration during vocalization [13], the channel that anchors our caller labels. Figure 1(a–Figure 1f) shows one released event across these aligned views.
Figure 1. V33DA overview and representation geometry. a. BirdPark rendering with calibrated microphones, camera poses, the 4-antenna FM receiver array, and triangulated bird positions. b. Synchronized multi-view frame with tracked individuals. c. Candidate-centered crop; the neon-green ring marks the vocalizing bird’s backpack. d. Accelerometer-derived vibration spectrograms; left color bars mark the blue, peach, and red birds. e. Per-antenna (A–D) radio telemetry—baseband signal power, relative phase, and compensation phase—over the event, with the same color bars. f. Five-channel microphone spectrograms. Panels a–f show the same verified event across geometry, video, on-body sensing, radio telemetry, and multichannel audio. g. UMAP of withheld on-body reference embeddings: the reference cleanly separates the caller from silent birds. h. SELDnet++ embeddings organized primarily by bird identity. i. SpatialScorer embeddings organized primarily by vocalizer position. Panels g–i preview the main result: transferable attribution depends on spatial organization more than on identity recognition.
Label construction and manual validation. Caller labels are anchored in the on-body vibration signal rather than in perceptual judgment. For each bird, a multichannel WhisperSeg model [22] proposes candidate windows from the demodulated on-body trace, and a sample is retained only when the event is visible in the microphone channels, the labeled bird shows the corresponding on-body vibration signature, and no overlapping call from another bird is present. Candidate state (detections, backbone-color identity, and triangulated 3D pose) is reconstructed from the synchronized video and radio streams (Appendix § 12.§ 3, § 12.§ 4, § 12.§ 6), and every candidate vocalization is manually reviewed against the full multimodal evidence (Appendix § 12.§ 7). The on-body vibration channel physically determines which bird produced each call; manual review therefore checks that the room audio, video, and 3D pose are consistent with that label, rather than inferring caller identity perceptually as in human active-speaker datasets. Of 39,329 reviewed candidates, 1,768 are rejected as invalid; reprojection filtering removes a further 152, and overlap filtering removes 3,784, yielding 33,625 released single-vocalizer events.
Release scope. The 33,625 events span BP01, BP02, and BP05, with 16,219 / 11,577 / 5,829 events per experiment (Appendix Table 8). All 10 released birds are distinct physical individuals, and no bird appears in more than one experiment. Birds are referenced by per-experiment backpack color codes, so a color reused across experiments refers to different physical individuals. Each event carries a per-event identifying the caller within the event’s up-to-four visible candidates; this index is stable within an experiment but experiment-local (full schema: Appendix § 12). Audio clips range from 40 to 784 ms (mean 150 ms, median 125 ms), and each sample includes 4–38 video frames with aligned radio telemetry.
V33DA++. V33DA++ is an unevaluated companion release to the main V33DA benchmark and ships three buckets; no result in this paper depends on it. overlap contains reviewer-verified calls dropped from V33DA because another bird vocalized in the same window, each tagged with the concurrent callers verified from their on-body vibration channels. padded contains ±2 s windows around V33DA events with original onsets and offsets preserved, supplying long-range context without altering the underlying event. overlap_padded applies the same s context to overlap events for long-context multi-caller tasks.
Benchmark Design
The main task is candidate-conditioned caller attribution. We evaluate every model under three progressively harder splits, designed so a model that has only memorised the training birds cannot look strong on all of them.
Evaluation regimes. Session-disjoint separates train, validation, and test by recording date while keeping all birds in every partition. This is the in-domain setting: a model sees every bird at training time and is asked to generalise across recording sessions of the same individuals, so scoring well requires nothing more than recognising familiar voices. Held-out experiment rotates the test experiment across BP01, BP02, and BP05 and trains on the remaining two. Because no bird appears in more than one experiment, every test event comes from an individual the model has never seen, and the shift also carries realistic cohort and recording-date differences. Leave-one-bird-out holds out positive vocalization examples for one bird within a single experiment while the remaining birds provide training and validation, and the cage, microphone geometry, and recording dates stay fixed. The held-out bird may still appear as a silent visible candidate in other birds’ events, where it is always labeled as a non-vocalizer, so LOO tests held-out-vocalizer generalization under fixed sensing conditions together with resistance to this negative prior; held-out experiment, where no bird is shared, is the cleaner transfer regime. This yields 4 folds for BP01 and 3 folds each for BP02 and BP05, for 10 folds in total. Table 2 summarises what varies, what is held fixed, and what each regime tests.
Table 2: Evaluation regimes in order of increasing difficulty. All three regimes evaluate the same task (candidate-conditioned caller attribution); they differ in which axes of variation are held out at test time.
| Regime | Held out at test | Held fixed | What it isolates |
| Session-disjoint | Recording date | Birds, cage, array | In-domain session noise |
| Held-out experiment | Bird cohort + acq. context | No birds shared | Identity + context shift |
| Leave-one-bird-out | Positive caller examples | Cage, array, cohort, dates | Held-out vocalizer |
Table 2.
Metrics. Top-1 attribution accuracy is the primary metric on every split. In leave-one-bird-out, the uniform fold mean is also per-bird recall because each test fold contains positive calls from one held-out physical bird; Appendix § 14.§ 4 reports the sample-weighted companion. 3D source localization is a secondary diagnostic, split into free localization, which regresses the 3D source position, and candidate-conditioned localization, induced from candidate scores or the selected candidate’s position [16, 46, 56]. Collapsing the two would conflate models that genuinely localise with models that only re-rank a known candidate set.
Reporting. For learned models, session-disjoint and held-out-experiment cells report mean standard deviation across training seeds. Leave-one-bird-out cells average seeds within each fold and then report mean standard deviation across the 10 folds, so fold-level variation stays visible rather than being smoothed away by seed averaging. Deterministic baselines are reported without . The withheld on-body reference is shown in every regime as a ceiling rather than as a competitor, because it uses the verification channel withheld from all other models.
Baselines
All baselines except the on-body reference solve the same task: given an event, choose which visible bird produced the vocalization. We group them into six families by how they turn the multimodal inputs into a per-candidate score, since the benchmark question is which formulation transfers rather than which backbone fits best in-domain. Full architectures and modality usage are detailed in Appendix § 13.
Chance and physics references. These rows establish how far a model can get without learned audio identity. Random picks one visible bird uniformly. LoudestMic attributes the call to the visible bird closest in 3D to the loudest microphone, with no time differences and no training. SRP [32] scores each candidate by how well the observed inter-microphone time differences match those expected from its location, again without voice identity or label training. SRP reranker layers a learned reranker on the same evidence, and NeuralSRP [42] replaces the handcrafted spatial features with a learned spatial representation while staying geometry-first.
Slot-indexed audio classifier. SELDnet [1] predicts a candidate slot index directly from the multichannel audio through a fixed 4-way head. Because the slot-to-bird mapping is stable within an experiment, on session-disjoint evaluation the head effectively learns per-bird audio signatures, while on leave-one-bird-out the held-out slot is absent from training by construction and the model has no mechanism to match audio against candidate state. This row represents the formulation that looks strong on familiar individuals but is structurally unable to generalise to unseen ones.
Candidate-scoring models. These score each visible bird from audio plus its candidate-side information (3D position, head direction, radio), so they can in principle identify unseen birds. SELDnet++ extends SELDnet with full candidate conditioning and is the only row using every candidate-side modality; SELDnet++ open removes the slot head to force pure candidate matching. SpatialScorer combines microphone-pair time-difference features with per-candidate geometry and cross-experiment regularisation. PairRank learns a pairwise preference between candidates, Contrastive embeds audio and each candidate’s spatial signature in a shared space and attributes by cosine similarity, and Whisper candidate [48] swaps the audio backbone for frozen Whisper features; Appendix § 14.11 also tests frozen bioacoustic front ends (AVES, BirdNET, animal2vec) under a candidate-scoring head.
Localization-first and literature-style spatial-audio baselines. These route attribution through localization, estimating a 3D source position and picking the nearest visible bird. AudioANN and RadioANN ship in three variants each (direct, locfirst, hybrid), so the effect of routing through localization is visible within a single backbone. ACCDOA [55], EINV2 [5], and CST-former [57] are literature-style array-audio architectures adapted only at the attribution head (Appendix § 13.12 reruns CST-former and Whisper candidate with the 60-epoch budget used for SELDnet), and BeamFusion adds learned per-candidate beamforming with transformer fusion.
Vision-only controls and withheld reference. PoseMotion scores candidates from 3D keypoint trajectories and VideoCandidate from per-bird camera crops, in the spirit of visible-speaker attribution in human video [51, 41] (Tao et al. 2021), measuring how much of attribution is already visible from video alone before audio and geometry combine. The withheld on-body reference uses the on-body vibration channel that anchors the labels and is reported as a ceiling rather than as a deployable baseline. Its performance measures the information available in the withheld verification channel and serves as a consistency check for the retained events.
Experiments
We evaluate all baselines under the three benchmark regimes on the same events and candidate sets. Learned rows report mean std across seeds; LOO entries average seeds within each fold before reporting mean std across folds, so fold heterogeneity is not smoothed away. The withheld on-body reference uses the verification channel and is reported only as a ceiling.
Attribution (Table 3). The best session model is the worst transfer model. SELDnet++ reaches 86.5% on session-disjoint evaluation but drops to 0% under leave-one-bird-out. SELDnet and AudioANN-direct fail in the same way. All three carry an identity-slot output head: at training time the model learns one slot per training bird, and when the test bird is unfamiliar, no slot fits.
| Method | Session | Held-out BP01 | Held-out BP02 | Held-out BP05 | Base | |||
| SRP (physics) | – | – | – | |||||
| SELDnet++ (slot) | ||||||||
| SpatialScorer | ||||||||
| ACCDOA | ||||||||
| EINV2 | ||||||||
| AudioANN-direct (slot) | ||||||||
| On-body ref. | – | – | – |
Table 3. Vocal attribution and LOO modality interventions across benchmark regimes (representative subset) using V33DA. Attribution accuracy (%) for one representative method per family; the full 24-method matrix is in Appendix Table 18. The last three columns are LOO accuracy changes (points) when one input channel is removed at test time on a single seed-42 checkpoint: zeroing the audio (), shuffling candidate geometry across events (), or zeroing the radio features (). Negative values indicate degradation. Session tests new recordings of known birds; Held-out BP01/BP02/BP05 rotate the test experiment; LOO mean averages within-experiment leave-one-bird-out folds. Learned rows report mean std across seeds. The withheld on-body reference is a ceiling.
The five rows that transfer (EINV2, SpatialScorer, ACCDOA, SRP, and AudioANN-locfirst) instead score the visible candidates directly, or attribute through localization. They include a classical physics method (SRP) and learned models, so the discriminating factor is the spatial formulation, not the choice of architecture. Their fine-grained ordering, however, is not resolved by ten LOO folds and three held-out experiments (Appendix § 14.§ 2).
The withheld on-body reference stays above 91% in every regime, indicating that the retained events are internally consistent under the withheld verification signal and leaving a substantial gap for the benchmark models. Figure 1(g–Figure 1i) shows the same contrast at the representation level. Appendix § 14.§ 1 gives the slot-collapse analysis, regime-level results, and sample-weighted LOO confirmations.
Close birds. Birds that sit close together leave the array a smaller timing margin. Stratifying LOO events by the distance from the vocalizer to its nearest competing candidate, ACCDOA falls from 73.0% above 50 cm to 46.6% below 10 cm, while random attribution changes by only 4.8 points (Appendix § 14.9). Even below 10 cm, a fixed GCC-PHAT decoder that uses no learned identity reaches 36.6% against 26.3% for random attribution, so close spacing is a measurable source of difficulty rather than a loss of spatial information.
Localization. We check whether the audio-spatial rows have learned a usable source position, rather than only a candidate ranking. Table 4 reports a representative subset; the full 16-row table is in Appendix Table 23. The results match the attribution picture: EINV2 and CST-former give the lowest errors in free 3D localization, and SpatialScorer and BeamFusion among candidate-conditioned methods. On LOO, SpatialScorer reaches 19.7 cm, about 7 cm below the random-candidate baseline (27.1 cm) and above the on-body reference (3.3 cm); this closes 31% of the range between the two, comparable to the 35% it closes in attribution accuracy (49.5% against 27.0% random and 91.3% for the reference). The same gap is even tighter on held-out BP05.
| Setting | Method | Session | Held-out BP01 | Held-out BP02 | Held-out BP05 | LOO mean |
| Free | EINV2 | |||||
| CST-former | ||||||
| Cand. | Random candidate | |||||
| SpatialScorer | ||||||
| BeamFusion | ||||||
| Ref. | On-body ref. |
Table 4. Localization across benchmark regimes (representative subset) using V33DA. Mean error in cm (lower is better). Setting: Free = unconstrained 3D regression; Cand. = candidate-conditioned (one of the visible birds). The full table is in Appendix Table 23. The random-candidate row picks one visible candidate uniformly at random; the shaded withheld on-body reference row is a ceiling for the candidate-conditioned setting only.
Modality interventions. The last three columns of Table 3 apply three test-time interventions to the transfer rows, each isolating one input channel: zeroing the audio, shuffling candidate geometry across events, and zeroing the radio features.
For ACCDOA and EINV2 the pattern holds on held-out BP01, held-out BP02, and LOO: removing audio costs 25–30 points and shuffling geometry costs 27–33 points, whereas zeroing the radio representation changes accuracy by less than a point, so transfer is carried by clean audio together with correct candidate geometry, with radio largely redundant once geometry is known. On held-out BP05, where every method is weakest, these audio and geometry effects shrink to roughly 7–12 points. SpatialScorer is a formulation-dependent exception on BP05, where zeroing radio costs 14.8 points. Position-noise robustness and cross-model significance tests (Appendix § 14.§ 2) split the rows along the same geometry-using vs geometry-agnostic line, confirming the central lesson: V33DA rewards usable spatial structure, not generic multimodality.
What V33DA reveals. High session-split accuracy does not imply transferable attribution. The matrix separates methods that exploit familiar individuals from methods that preserve usable spatial structure for unseen individuals, and four observations make this concrete. Familiar-voice recognition is effective when every test bird is familiar: SELDnet++ reaches 86.5% session-disjoint. The same formulation does not reach held-out vocalizers: with an identical encoder and training, AudioANN reaches 0.0% LOO when it attributes directly and 53.0% when it attributes through localization. Candidate conditioning alone is not sufficient: SELDnet++ open, PairRank, Whisper candidate, and CST-former all remain below the 27.0% random reference on LOO. And the rows that transfer rely on the array: removing microphones lowers SpatialScorer from 51.2% to 26.0% LOO, and permuting microphone channels lowers ACCDOA from 64.3% to 38.8% session-disjoint (Appendix § 14.10). The strongest transfer rows share a formulation, not a backbone: they score visible candidates directly, reason over spatial geometry, or route attribution through localization. Multi-individual attribution should therefore be evaluated as transfer rather than held-out-date accuracy on familiar individuals, identity shift should be separated from broader experiment shift when possible, and a model’s actual modality use should be checked directly rather than inferred from headline accuracy.
The on-body channel serves here to construct trustworthy labels, not as a permanent recording requirement. Attribution from room sensors is most needed where voice identification of familiar individuals breaks down: juvenile subsong changes rapidly and carries little individual signature, rare vocalizations such as copulation calls may be too sparse to learn, and new birds enter a cohort without labeled calls. Acoustic clustering does not remove this need: even with segmented events, the true number of birds, and an encoder trained on the same cohort, -means reaches NMI 0.61 and ARI 0.58, and the clusters must still be mapped to the tracked candidates (Appendix § 14.11).
Limitations
V33DA has three scope boundaries. First, all data come from a single sensing installation with fixed cage, microphone, camera, and 4-antenna geometry, covering 3 experiments, 10 birds, and 33,625 verified vocalizations; cross-installation transfer is outside the scope of our regimes. Second, the per-bird radio transmitters lack a frequency-stabilizing quartz, so the carrier drifts faster than the released analysis window; the 12-dimensional radio summary captures only the standard deviation of the carrier, not its fast temporal structure, and prior work [52] has shown these sub-frame-rate dynamics encode fast movements such as wingflaps. The apparent redundancy of radio for localization (Table 3) likely reflects this feature summary, not the modality itself. Third, the V33DA benchmark covers only verified single-vocalizer events with stable candidate state; multi-caller attribution, source separation, vocal activity detection, and longer-context tasks are left to V33DA++, which is released as an unevaluated companion (§ 3).
Conclusion
We introduced V33DA, a physically verified multimodal benchmark for spatial vocal attribution in social zebra finches. Across session-disjoint, held-out-experiment, and leave-one-bird-out evaluation, V33DA shows that high in-domain accuracy can come from bird-identity shortcuts, while transfer to unseen individuals requires spatial reasoning. Slot-indexed audio classifiers look strong in-domain but collapse under unseen birds; candidate-aware spatial and localization-first methods transfer, but remain well below the withheld on-body reference, which measures the ceiling available when the withheld verification channel is observed. Beyond caller attribution, the same synchronized sensor state supports audio, video, pose, and radio modeling.
Acknowledgments and Disclosure of Funding
This work was supported by the Swiss National Science Foundation (grant 31003A_182638) and the NCCR Evolving Language, Swiss National Science Foundation Agreement No. 51NF40_180888, and by a collaborative project with the Swiss Data Science Center (SDSC). The funders had no role in study design, data collection, analysis, or manuscript preparation. The authors declare no competing interests.
Ethics, Licensing, and Release
All animal procedures were approved by the Cantonal Veterinary Office of Zurich under license ZH148/2020 and comply with Swiss and EU animal welfare regulations. The birds were housed in enriched environments with ad libitum food and water.
The benchmark can support research on animal communication, multimodal learning, and automated behavioral analysis under controlled, physically verified supervision. We do not currently foresee material societal risks from releasing V33DA itself. A remote concern is that attribution or localization systems benchmarked in this setting could eventually be repurposed for surveillance-style monitoring in other contexts, but that possibility is far removed from the present dataset, task, and intended use.
The dataset is released under a CC-BY-4.0 license at https://huggingface.co/datasets/songbirdini/v33da, with the evaluated V33DA++ companion release at https://huggingface.co/datasets/songbirdini/v33da_pp. The accompanying code and baseline implementations are available at https://github.com/marisbasha/v33da. External software and pretrained components used in preprocessing and comparison, including WhisperSeg/Whisper, YOLOX/YOLO-Pose, and OC-SORT, are credited in the paper and remain subject to their original licenses and terms of use. The dataset contains no personally identifiable information.
Artifact availability. The submission packet includes the public V33DA dataset URL, the executable code URL, and a Croissant JSON file describing the main V33DA benchmark artifact. V33DA++ is linked as an unevaluated companion release of overlap and padded-context files.
Task Coverage of Related Resources
Dataset Construction Pipeline
This appendix documents the full curation pipeline that converts raw BirdPark recordings [52] into the released V33DA events. We describe raw acquisition, candidate proposal from on-body sensing, vision-based state reconstruction, identity assignment, 3D triangulation, manual verification, and post-review quality filtering. For each stage we specify the inputs, the processing steps, and the criteria by which events are retained or removed, so that the release is auditable end to end.
Raw Data Acquisition
Each recording session produces a 7-minute H5 file containing:
daqSignals: float32: 7 DAQ channels (5 cage microphones + 2 nest microphones) at 24,414 Hz. BP05 additionally logs a room reference microphone, giving ; only the 5 cage microphones are used in the benchmark.radioSignals: float32: FM-demodulated instantaneous transmitter frequency from body-mounted transmitters at 24,414 Hz.radioFrameSignals: 21 per-bird per-frame features (tracking frequency, master- and rotated-vector phase angles, per-antenna compensation phases, and baseband signal powers and variances) tracked at video frame rate (47.68 Hz).Synchronized composite video (2976×2016, 47.68 fps) combining top, back, and side camera views.
Table 6 summarizes the raw recording volumes. In total, approximately 74 hours of continuous multi-modal recordings were processed to extract the 33,625 vocalization events in the released dataset.
| Experiment | Days | H5 Files | Recording Hours |
| juvExpBP01 | 10 | ||
| juvExpBP02 | 3 | ||
| juvExpBP05 | 3 |
Table 6. Raw data volumes per experiment.
Vocalization Segmentation
Vocalization candidates are proposed from the body-mounted accelerometer-derived signals using a multichannel WhisperSeg model [22, 48], a Whisper-large variant fine-tuned for bioacoustic segmentation. WhisperSeg is used only to propose candidate windows; it does not define final benchmark labels. For each bird, the model receives three input channels:
the target bird’s radio/accelerometer signal,
the sum of the other birds’ radio/accelerometer signals,
the sum of the cage-microphone signals.
Each channel is peak-normalized to , resampled to 16 kHz, converted to an 80-bin mel spectrogram, concatenated to form a 240-bin representation, and interleaved back to 80 bins before being processed by the Whisper encoder-decoder. The model predicts onset and offset times for each candidate vocalization. The summed cage-microphone channel is used only as a shared timing and event-presence cue; final caller identity is established downstream by comparing the target bird’s on-body vibration signature to those of the other birds, not by localizing the call from room acoustics alone. Candidate windows are discarded if the event is not visible in the microphone channels, if the on-body evidence is ambiguous, or if overlapping vocalizations are present. Upstream misses reduce coverage of the released event set but do not corrupt the labels of retained events: V33DA contains only candidate windows that survive manual verification and post-review filtering.
2D Detection and Keypoint Estimation
Bird detection and keypoint estimation. YOLOXPose [19, 35] performs joint bird detection and five-keypoint estimation in each camera view. The model runs full-frame inference at input resolution with a detection score threshold of 0.8 and NMS threshold of 0.7, producing per-bird bounding boxes and five 2D keypoints: beak tip, head center, backpack center, tail base, and tail tip. Separate YOLOXPose checkpoints (Tiny variant) are trained for each camera view (top, back, side) on a pool of manually annotated frames collected across multiple BirdPark recording sessions (including but not limited to the three experiments in V33DA). Each annotated frame includes five keypoints per visible bird with visibility flags and color-coded identity labels. Because camera placement varies between experiments, we fine-tune the detector per-experiment using the subset of annotations from that experiment’s camera configuration.
2D tracking. OC-SORT [4] maintains consistent track identities across frames within each camera view independently, with a history buffer of 500 frames.
Identity Assignment via Backpack Color Detection
Each bird wears a uniquely colored backpack. Identity is assigned within each vocalization window from the tracked top-view detections using the color stage in the BirdPark tracking pipeline.
Stage 1: Frame selection. Color inference is run only on the top view and only on selected frames within the vocalization window. The tracker first identifies informative frames around track starts, ends, and long gaps, then adds periodic samples between those events. Within each sampling window, the pipeline picks the frame with the best geometric support, favoring frames with visible backpack keypoints and good 2D bird separation.
Stage 2: Color detection and track assignment. On each selected frame, a K-NN color classifier assigns backpack color from a pre-built gallery of labeled patches. The classifier extracts a patch in CIELAB color space centered on the backpack keypoint, flattens it, and queries the nearest neighbors from the gallery (approximately 4,700 labeled patches per experiment) using distance. The majority vote determines the predicted color. A 1-second temporal smoothing window (47 frames at the native frame rate) and a minimum run length of 3 seconds suppress spurious predictions. Per-frame assignments with confidence below 0.5 are discarded. The result is a set of per-frame color observations for each top-view track.
Stage 3: Temporal stabilization. Per-frame color observations are converted into track-level identities with confidence pre-filtering and hysteresis segmentation. The dominant color segment for each track is retained, and the resulting color label is mapped to bird identity using the experiment metadata. Final dataset release does not depend on this automatic step alone: all released events are manually reviewed, and candidates with uncertain or inconsistent identities are excluded from release.
Pipeline Component Evaluation
Color classification accuracy. Table 7 reports the K-NN backpack color classification accuracy on held-out annotated frames for the three V33DA experiments. Per-experiment galleries are built exclusively from each experiment’s own annotated frames to prevent cross-experiment color confusion. Patches with occluded backpacks (COCO visibility ) are excluded from both the gallery and evaluation set. Each clean patch is augmented (4 rotations 2 flips 3 brightness offsets 2 blur levels) before gallery construction. Outlier removal via leave-one-out K-NN consensus further filters mislabeled or ambiguous patches.
| Experiment | Clean patches | After aug | Accuracy |
| juvExpBP01 | 2,000 | 96,000 | 98.1% |
| juvExpBP02 | 1,050 | 50,400 | 95.2% |
| juvExpBP05 | 1,040 | 49,920 | 97.3% |
| Average | 96.9% |
Table 7. K-NN backpack color classification accuracy per experiment.
Ablation: gallery construction. Averaging across 15 BirdPark experiments (including but not limited to the three V33DA experiments), color accuracy improves from 88.9% (shared gallery, baseline) to 91.2% (per-experiment galleries), 93.0% (+ outlier removal), 94.8% (+ occlusion filtering), and 96.4% (+ augmentation).
Keypoint quality. Keypoint quality is assessed indirectly through 3D-to-2D reprojection error after triangulation. Median reprojection errors on released samples are 7.9 px (BP01), 2.5 px (BP02), and 12.3 px (BP05), corresponding to roughly 1–2 mm spatial precision at cage scale. Samples with reprojection error px are excluded (see Section B.8).
Cross-View Assignment and 3D Triangulation
Camera calibration. Per-experiment camera calibrations are computed from manually annotated 3D reference points on the cage structure (cage corners, perch endpoints, microphone positions), following standard multi-view geometry practice [27]. We use cv2.calibrateCamera() to estimate intrinsic matrices , distortion coefficients, and extrinsic parameters (rotation , translation ) for each view. Calibration reprojection error on reference points is 2.5–3.7 px across experiments.
Cross-view matching. To establish correspondence between 2D detections across the top and back camera views, we compute the Sampson distance between each pair of detections using the fundamental matrix derived from the calibrations [27]. For each vocalization event, we enumerate all possible top–back detection pairs and select the assignment that minimizes total Sampson distance while maintaining geometric consistency.
Triangulation. Matched 2D keypoint pairs are triangulated using the Direct Linear Transform (DLT) method [27]. Given a 2D point in view 1 and in view 2, we construct the system:
where is the -th row of the projection matrix . The 3D point is recovered via SVD. Triangulation does not use lens distortion (distCoeffs=None) as the calibration reference points span a limited field of view where distortion effects are minimal.
Coordinate system. All 3D coordinates are expressed in -up convention (millimeters): = left–right, = front–back, = up–down. The coordinate frame is defined by the camera calibration reference points.
Human Verification
All candidate windows are manually reviewed by a trained annotator using the interface in Figure 2. For each candidate, the reviewer inspects:

Figure 2. Verification interface used during dataset curation. (a) Experiment selector. (b) Synchronized video frame with overlaid keypoints and color-coded identities; the selected vocalizer is highlighted by a green ring. (c) Reviewer controls for navigating events, selecting the vocalizer identity, and assigning the review verdict; the crop strip shows the currently visible tracks. (d) Per-bird accelerometer spectrograms; the selected bird shows the vocalization-associated vibration signature. (e) Five-channel microphone spectrograms aligned to the same event.
Video and keypoints: to confirm that visible birds are correctly tracked and assigned to the correct backpack identity;
Microphone audio: to confirm that a vocalization is present at the expected time;
On-body sensing and radio telemetry: to confirm that the labeled bird shows the characteristic thoracic-vibration signature while the event is simultaneously visible in the room microphones.
Candidates are excluded if the automatic pipeline introduces identity swaps, missed detections, triangulation failures, ambiguous on-body evidence, or overlap with another bird’s vocalization. The review stage therefore functions as a multimodal consistency check over synchronized evidence rather than as a subjective labeling step. Retained events have confirmed (a) vocalizer identity, (b) keypoint-to-bird assignment for all visible birds, and (c) temporal consistency across video, microphone audio, and on-body sensing.
Quality Filtering and Validation
After manual review, valid candidates pass through a deterministic quality filter before entering the release:
Temporal alignment. Audio, video, radio, and accelerometer streams must agree within the ms event extraction window.
3D plausibility. The triangulated vocalizer position must lie within the cage bounds (, , mm).
Reprojection error. The vocalizer’s 3D backpack keypoint is reprojected into the top and back views using
cv2.projectPoints(). Candidates with reprojection error above 40 px in either view are discarded. Median reprojection errors on released events are 7.9 px (BP01), 2.5 px (BP02), and 12.3 px (BP05); the 40 px threshold corresponds conservatively to about 1–2 cm of worst-case spatial error.2D displacement. The maximum frame-to-frame beak displacement must not exceed 40 px in either top or back view, ensuring limited motion during the short event.
Identity consistency. The color-based identity assignment must remain consistent across the event window.
Of the 39,329 reviewed candidates, 1,768 are rejected during review and 37,561 are marked valid. The geometric and overlap filters then remove 152 additional candidates for excessive reprojection error and 3,784 for residual temporal overlap, yielding 33,625 released events.
| Dataset summary | Release audit | ||||||
| Experiment | Birds | Samples | Days | Dates | Stage | Count | Share |
| BP01 | 4 | 16,219 | 10 | Jul–Aug ’21 | Reviewed | 39,329 | 100.0% |
| BP02 | 3 | 11,577 | 3 | May ’22 | Valid | 37,561 | 95.5% |
| BP05 | 3 | 5,829 | 3 | Jul ’23 | Invalid | 1,768 | 4.5% |
| Total | 10 | 33,625 | 16 | Reproj. filtered | 152 | 0.4% | |
| Overlap filtered | 3,784 | 10.1% | |||||
| Released | 33,625 | 89.5% |
Table 8. Dataset summary and release audit. Left: released events by experiment. Right: reviewed = valid + invalid; valid is then partitioned into reprojection-filtered, overlap-filtered, and released samples.
Post-filtering Statistics
| Metric | BP01 | BP02 | BP05 |
| Samples | 16,219 | 11,577 | 5,829 |
| Reproj error (median, px) | 7.9 | 2.5 | 12.3 |
| 3D displacement (median, mm) | 2.9 | 2.5 | 2.5 |
| 2D disp top (median, px) | 4.0 | 4.3 | 4.6 |
| 2D disp back (median, px) | 4.4 | 3.5 | 3.6 |
Table 9. Quality metrics after filtering.
Audio Extraction
For each vocalization event, we extract the full audio from onset to offset plus a ms padding before and after from the 5 cage microphones (DAQ channels 0–4). The audio is stored as 5-channel WAV files (float32, 24,414 Hz), with clip durations ranging from 40–784 ms (mean: 150 ms, median: 125 ms). Nest microphones (channels 5–6) are excluded as they record inside enclosed nest boxes and do not carry useful spatial information for attribution.
Radio Telemetry Extraction
The 21 per-bird per-frame radio signals are extracted for the video frames within the vocalization window. Following BirdRadio’s definitions [52], the signals are: PLL tracking frequency (frq); master-vector phase angle (thetaS) and per-antenna rotated-vector angles (thetaA–D); per-antenna compensation phases (phiA–D) and their sum (phiM); and the per-antenna baseband signal powers (powRA–RD, powRM) and variances (powNA–ND, powNM), all in dBFS. BirdRadio does not compute an angle of arrival. For the 11 samples where radio frame counts fell 1 frame short of the video frame count, the last radio frame is repeated to maintain alignment.
Accelerometer Extraction
The raw FM-demodulated radio signal per bird is extracted at audio sample rate (24,414 Hz) for the same ms window as the audio. This signal is the physical basis for ground-truth labeling: when a bird vocalizes, thoracic vibrations modulate the FM carrier frequency, producing a distinctive waveform in the demodulated signal. The accelerometer column is provided for label verification and is withheld from benchmark inputs.
Dataset Characterization
Per-bird vocalization counts. Table 10 reports the number of vocalizations per bird in each experiment and session-disjoint split. Class imbalance is substantial: within BP01, Bird 3 still contributes the largest share of samples, while within BP05 one bird contributes only 56 events in the entire release. This imbalance is one reason we report per-fold and per-experiment breakdowns rather than relying only on pooled means.
| Experiment | Bird | Train | Val | Test |
| BP01 | Bird 0 | 2,827 (21.9%) | 218 (21.6%) | 918 (39.4%) |
| BP01 | Bird 1 | 1,694 (13.2%) | 197 (19.5%) | 386 (16.6%) |
| BP01 | Bird 2 | 3,530 (27.4%) | 248 (24.6%) | 466 (20.0%) |
| BP01 | Bird 3 | 4,830 (37.5%) | 345 (34.2%) | 560 (24.0%) |
| BP02 | Bird 0 | 726 (16.3%) | 844 (20.3%) | 495 (16.7%) |
| BP02 | Bird 1 | 1,870 (42.0%) | 1,540 (37.1%) | 1,179 (39.7%) |
| BP02 | Bird 2 | 1,859 (41.7%) | 1,771 (42.6%) | 1,293 (43.6%) |
| BP05 | Bird 0 | 753 (37.1%) | 697 (40.9%) | 690 (33.0%) |
| BP05 | Bird 2 | 21 (1.0%) | 26 (1.5%) | 9 (0.4%) |
| BP05 | Bird 3 | 1,255 (61.9%) | 983 (57.6%) | 1,395 (66.6%) |
Table 10. Per-bird vocalization counts by experiment and session-disjoint split. Percentages indicate within-experiment fractions.
Leave-one-bird-out fold sizes. Table 11 provides per-fold sample counts for all three experiments and all released birds.
| Experiment | Held-out bird | Train | Val | Test | Test % |
| BP01 | Bird 0 | 10,961 | 1,295 | 3,963 | 24.4% |
| BP01 | Bird 1 | 12,535 | 1,407 | 2,277 | 14.0% |
| BP01 | Bird 2 | 10,763 | 1,212 | 4,244 | 26.2% |
| BP01 | Bird 3 | 9,418 | 1,066 | 5,735 | 35.4% |
| BP02 | Bird 0 | 8,569 | 943 | 2,065 | 17.8% |
| BP02 | Bird 1 | 6,274 | 714 | 4,589 | 39.6% |
| BP02 | Bird 2 | 5,989 | 665 | 4,923 | 42.5% |
| BP05 | Bird 0 | 3,329 | 360 | 2,140 | 36.7% |
| BP05 | Bird 2 | 5,201 | 572 | 56 | 1.0% |
| BP05 | Bird 3 | 1,974 | 222 | 3,633 | 62.3% |
Table 11. Leave-one-bird-out fold sizes for all experiments.
Inter-bird spatial separation. Table 12 reports per-pair separation statistics for all three experiments. In BP01, Birds 0 and 1 are frequently co-located (25.4% within 100 mm), as are Birds 2 and 3 (17.5%). BP02 has wider overall separation, with the closest pair (Birds 1–2) still averaging 346 mm. BP05 contains one comparatively ambiguous pair, Birds 2–3 (11.7% within 100 mm), while the other BP05 pairs remain substantially farther apart. This co-occurrence structure directly explains the per-fold SRP variance in the LOO evaluation.
| Experiment | Bird pair | Mean (mm) | Median (mm) | % < 100 mm |
| BP01 | 0–1 | 228 | 199 | 25.4% |
| BP01 | 0–2 | 407 | 411 | 0.6% |
| BP01 | 0–3 | 395 | 406 | 0.4% |
| BP01 | 1–2 | 415 | 424 | 0.4% |
| BP01 | 1–3 | 410 | 412 | 0.3% |
| BP01 | 2–3 | 254 | 236 | 17.5% |
| BP02 | 0–1 | 427 | 395 | 1.4% |
| BP02 | 0–2 | 451 | 420 | 0.4% |
| BP02 | 1–2 | 346 | 328 | 6.4% |
| BP05 | 0–2 | 410 | 387 | 2.1% |
| BP05 | 0–3 | 430 | 409 | 0.3% |
| BP05 | 2–3 | 299 | 279 | 11.7% |
Table 12. Pairwise inter-bird distances for all experiments.
Per-fold ceiling estimates. Table 13 reports per-fold ceiling estimates for all experiments, computed by adding isotropic Gaussian noise () to the true vocalizer position and assigning to the nearest candidate bird. BP02 ceilings are substantially higher than BP01 and BP05 due to wider inter-bird separation.
| Experiment | Held-out bird | mm | mm | mm |
| BP01 | Bird 0 | 94.5% | 79.4% | 66.4% |
| BP01 | Bird 1 | 91.0% | 74.9% | 62.6% |
| BP01 | Bird 2 | 94.8% | 82.3% | 69.9% |
| BP01 | Bird 3 | 94.8% | 82.5% | 70.3% |
| BP02 | Bird 0 | 98.9% | 91.7% | 83.0% |
| BP02 | Bird 1 | 97.4% | 88.8% | 78.4% |
| BP02 | Bird 2 | 98.2% | 89.8% | 79.7% |
| BP05 | Bird 0 | 94.5% | 80.8% | 68.5% |
| BP05 | Bird 2 | 91.3% | 78.4% | 66.9% |
| BP05 | Bird 3 | 95.2% | 81.9% | 69.5% |
Table 13. Per-fold ceiling estimates for LOO across all experiments.
Overlap exclusion. Of the 37,561 strict valid candidates entering the final geometric release stage, 3,784 are removed by overlap filtering. Temporal overlap is therefore the dominant explicit exclusion block within this final stage, and because WhisperSeg proposes candidate windows independently from each bird’s accelerometer channel, this overlap exclusion is approximately uniform across birds within each experiment.
Retained vs. excluded candidate bias. We compared the retention behaviour across birds within each experiment to check whether the curation pipeline is systematically biased. Per-bird retention varies by at most 1–6 percentage points across birds within the same experiment: for BP01 the four birds’ retention range is 5.7–6.9%, for BP02 it is 12.8–18.5%, and for BP05 it is 26.5–28.3%.
No single bird is disproportionately excluded. The per-experiment retention rate itself varies more substantially (BP01 6.4%, BP02 16.1%, BP05 27.6%), because WhisperSeg over-proposes more aggressively on the longer BP01 recordings (321,918 upstream candidates in BP01 vs. 90,108 in BP02 and 24,906 in BP05, from which the 16,219 / 11,577 / 5,829 release sets are ultimately retained); the validation pipeline is what corrects for this.
Baseline Implementation Details
This appendix provides full architectural, training, and evaluation details for all baselines reported in the main paper.
Modality Usage Summary
Table 14 summarizes which input modalities each baseline consumes at inference and the attribution strategy it uses.
Table 14: Baseline modality usage and attribution strategy. Audio: 5-channel microphone array. 3D: triangulated bird beak position. Head: head orientation vector. Radio: per-bird FM radio telemetry (summary and/or temporal). Video: synchronized RGB crop tubes. Accel: on-body accelerometer (withheld reference only). A checkmark indicates the modality is consumed at inference; (opt) indicates the model has a variant with and without that input.
| Family | Model | Audio | 3D | Head | Radio | Video | Accel | Attribution |
| Physics | LoudestMic | ✓ | ✓ | – | – | – | – | Nearest to loudest mic |
| SRP ±20 | ✓ | ✓ | – | – | – | – | Nearest to SRP peak | |
| SRP Reranker | ✓ | ✓ | ✓ | ✓ | – | – | Learned reranker | |
| NeuralSRP | ✓ | ✓ | – | (opt) | – | – | Candidate scorer | |
| Slot-indexed | SELDnet | ✓ | – | – | – | – | – | 4-way slot head |
| Candidate-centric | SELDnet++ | ✓ | ✓ | ✓ | ✓ | – | – | Hybrid direct + cand |
| SELDnet++ open | ✓ | ✓ | ✓ | ✓ | – | – | Candidate-only matching | |
| SpatialScorer | ✓ | ✓ | – | (opt) | – | – | TDOA ⊙ audio scorer | |
| PairRank | ✓ | ✓ | ✓ | ✓ | – | – | Pairwise ranking | |
| Contrastive | ✓ | ✓ | – | (opt) | – | – | Cosine similarity | |
| Whisper candidate | ✓ | ✓ | ✓ | ✓ | – | – | Candidate scorer | |
| SELD & loc-first | ACCDOA | ✓ | ✓ | (opt) | (opt) | – | – | |
| EINV2 | ✓ | ✓ | (opt) | (opt) | – | – | Track–candidate dist | |
| CST-former | ✓ | ✓ | ✓ | ✓ | – | – | Candidate scorer | |
| BeamFusion | ✓ | ✓ | – | ✓ | – | – | Beam + transformer | |
| AudioANN ×3 | ✓ | ✓ | – | – | – | – | Direct / loc / hybrid | |
| RadioANN ×3 | – | ✓ | – | ✓ | – | – | Direct / loc / hybrid | |
| Vision | PoseMotion | – | ✓ | ✓ | – | – | – | Per-bird motion score |
| VideoCandidate | – | ✓ | ✓ | – | ✓ | – | Per-bird crop score | |
| Reference | On-body ref. | – | ✓ | – | – | – | ✓ | Per-bird vibration |
Table 14.
Physical Constants and Data Format
Table 15: Recording system parameters and physical constants.
| Parameter | Value |
| Audio sample rate | 24,414.0625 Hz |
| Video frame rate | 47.6837158203125 Hz (= 390625/8192) |
| Samples per video frame | 512 |
| Speed of sound | 343,000 mm/s (20°C) |
| Cage dimensions (X × Y × Z) | 1246 × 896 × 1294 mm |
| Cage bounds X (left–right) | [−90.8, 1155.2] mm |
| Cage bounds Y (front–back) | [−1563.9, −667.9] mm |
| Cage bounds Z (bottom–top) | [236.0, 1530.0] mm |
| Coordinate system | Z-up (mm) |
| Number of cage microphones | 5 (Rode NT1, ∼17 dB gain) |
| Number of nest microphones | 2 (excluded from dataset) |
| Number of radio channels | 4 per experiment |
| Radio tracking range | ±750 / ±700 / ±900 kHz (BP01 / BP02 / BP05) |
| Radio gain | 68 dB |
| Local oscillator frequency | 325 / 350 / 340 MHz (BP01 / BP02 / BP05) |
Table 15.
Audio clip statistics. Clip durations range from 40–784 ms (mean: 150 ms, median: 125 ms) at 24,414 Hz. Each clip contains the full vocalization from onset to offset plus ±10 ms padding.
Video frame statistics. Each sample includes 4–38 video frames (mean: 9.1) from the composite multi-view video (2976 × 2016 pixels, 47.68 fps).
Position normalization. All 3D positions are normalized to [0, 1] by: , where and are the cage bounds listed above. Denormalization reverses this mapping.
Bird masking. BP01 and BP05 have 4 birds; BP02 has 3 birds. The data loader pads arrays to the maximum bird count in each batch, with a boolean mask indicating valid birds. Padded positions, head orientations, and radio features are set to zero; logits at padded positions are set to before softmax.
Radio feature summary. The 12-dimensional per-bird radio summary consists of:
Mean and standard deviation (over frames in the window) of the baseband signal power (dBFS) at the four antennas and the summed master vector:
powRA,powRB,powRC,powRD,powRM(10 features).Standard deviation of the PLL tracking frequency
frq(1 feature).Standard deviation of the master-vector phase angle
thetaS(, in degrees; 1 feature).
All 21 raw radio signals per bird per frame are available in the parquet files for custom feature extraction. Using BirdRadio’s naming Rüttimann et al. [ [52], the 21 signals are: the PLL tracking frequency frq; the master-vector phase thetaS () and the per-antenna rotated-vector angles thetaA–thetaD (); the per-antenna compensation phases phiA–phiD () and their sum phiM; the baseband signal powers powRA–powRD,powRM and variances powNA–powND,powNM (all in dBFS, at the sender frequency). None of these is an angle of arrival.
Classical Baselines (No Training)
Random baseline. For each sample, the predicted vocalizer is drawn uniformly at random from the candidate birds present in the scene. Expected accuracy: (25% for 4 birds, 33% for 3 birds).
LoudestMic. LoudestMic is a deterministic energy-plus-geometry heuristic. For each event, per-channel root-mean-square energy is computed over the audio window and the loudest microphone is identified:
where is the waveform on mic . The predicted vocalizer is the visible bird whose 3D position is closest to the loudest microphone’s 3D position :
No time-difference information, voice identity, or training is used. This baseline sits between Random and SRP in the deterministic-spatial family: it exploits which mic heard the call loudest (a proxy for proximity) but, unlike SRP, it cannot resolve configurations where multiple birds are similarly close to the same microphone.
SRP ±20. Steered Response Power with Phase Transform (SRP-PHAT) [11] is our strongest classical baseline. For each of the microphone pairs, we compute the GCC-PHAT cross-correlation [32]:
where is the FFT of the -th microphone channel and .
For each candidate bird at 3D position , we compute the expected TDOA for each microphone pair :
where is the -th microphone position and mm/s is the speed of sound.
The SRP score for bird is:
where samples (820 s) is the search window that compensates for reverberant TDOA bias. The predicted vocalizer is .
SRP requires no training. It uses the known 3D microphone positions (Section 3.1) and the candidate bird positions from the vision pipeline. The search window was selected based on analysis of the GCC-PHAT peak distribution (see main paper ceiling estimate).
Audio preprocessing: each channel is zero-mean and unit-variance normalized before FFT.
SELDnet (Audio-Only Learned Baseline)
SELDnet is a convolutional-recurrent audio pipeline adapted from the original SELDnet formulation [1] and later DCASE SELD baselines [46] to event-local candidate attribution.
Label space. SELDnet is a slot-indexed attribution model. Its output head predicts one of up to four event-local candidate slots, corresponding to the vocalizer_idx field in the release. This index is not a global bird identity. Within a given experiment the slot-to-bird mapping is stable across events, so a slot classifier can effectively memorize the vocal signatures of that experiment’s birds. Across experiments the same slot index refers to different physical birds, and in within-experiment leave-one-bird-out evaluation the held-out bird’s slot is absent from training entirely. A pure slot classifier therefore fails by construction under LOO, which explains the zero-accuracy behavior of SELDnet and other slot-indexed baselines.
Architecture.
Input: 5-channel mel spectrogram ().
Mel extraction: 80-bin log-mel spectrogram at 24,414 Hz, , hop length 256 samples, Hz, Hz.
CNN: Three convolutional blocks, each consisting of Conv2d BatchNorm2d ReLU MaxPool2d(4,1). Channel progression: 5 64 128 256. Kernel size: , padding 1. Pooling reduces the frequency dimension by per block (80 20 5 1).
GRU: 2-layer bidirectional GRU with hidden size 128, yielding 256-dim output per time step. Dropout 0.1 between layers.
Pooling: Mean pooling over the time dimension.
Slot-indexed attribution head: Linear(256, 128) ReLU Dropout(0.2) Linear(128, ), where is the maximum number of visible candidates in any event. At inference a per-event candidate mask zeroes out slot logits for unused positions.
DOA head: Linear(256, 64) ReLU Linear(64, 3) Sigmoid (used as an auxiliary source-position head).
Parameters. 1.03M total.
Audio preprocessing. Each channel is independently zero-mean, unit-variance normalized. Mel spectrograms are instance-normalized: per-sample, per-channel zero-mean and unit-variance over frequency and time dimensions.
PairRank and SpatialScorer
PairRank. PairRank removes the fixed class head and instead learns pairwise candidate preferences. The model first computes GCC-PHAT summary statistics for each of the 10 microphone pairs (peak, mean, and sharpness), producing a 30-dimensional audio summary. This summary is mapped through an MLP to a 96-dimensional audio embedding. Each candidate bird is encoded from normalized 3D position (3), head orientation (3), radio summary (12), and expected TDOA values for the 10 microphone pairs, again using a 96-dimensional MLP. For every candidate pair , the model forms a pair feature containing the two candidate embeddings, their relative displacement, their Euclidean distance, and the mean difference in expected TDOA. A pairwise scorer predicts whether bird is more plausible than bird , and the aggregated pairwise wins become the final candidate scores. Localization is candidate-conditioned: the predicted position is the softmax-weighted average of candidate positions.
SELDnet++. SELDnet++ extends SELDnet with full multimodal candidate conditioning. The audio encoder is identical to SELDnet (5-channel mel-spectrogram 3-layer CNN with frequency pooling 2-layer BiGRU temporal mean pool 256-dim embedding). Each candidate bird is encoded from the concatenation of its 3D position (3), head orientation (3), radio summary (12), and a radio temporal encoding (7-channel Conv1d BN ReLU Conv1d BN ReLU AdaptiveAvgPool1d 64), yielding an 82-dimensional input that is mapped through a 2-layer MLP to a 128-dimensional candidate embedding. The audio embedding is projected to 128 dimensions, and per-candidate scores are computed as (normalized dot product). A slot-indexed classification head (Linear ReLU Dropout Linear) with the same 4-way output as SELDnet is blended with the candidate scores via a learnable sigmoid gate , initialized at 0.5; SELDnet++ open zeros this gate and keeps only the candidate-scoring branch, forcing attribution to match the audio embedding against candidate state without any slot-indexed shortcut. SELDnet++ is the only model in the suite that combines the SELDnet audio backbone with all four candidate-side modalities (position, orientation, radio summary, and radio temporal).
SpatialScorer. SpatialScorer is the strongest purely spatial candidate model in the current suite. It uses GCC-PHAT lag features only, intentionally excluding mel spectrograms and timbre-heavy audio summaries. The 10 microphone-pair cross-correlations are cropped to a sample lag window and stacked into a tensor. A CNN encoder maps this GCC volume to a 256-dimensional audio embedding. In parallel, each candidate bird is encoded from its expected TDOA vector across the same 10 microphone pairs, and optional radio context is obtained from a temporal radio encoder. Candidate scores are produced from the concatenation of the audio embedding, candidate TDOA embedding, their elementwise product, and optional radio features. We additionally use a gradient-reversal adversarial head [18] to discourage experiment-specific shortcuts, making this baseline explicitly transfer-oriented rather than slot-indexed.
Localization-First AudioANN and RadioANN Baselines
AudioANN family. AudioANN uses a shared audio backbone built from a 64-bin multichannel mel front-end, a 3-layer CNN ( channels), and a 2-layer bidirectional LSTM with hidden size 128. The pooled audio representation is used in three ways:
AudioANN-direct: a direct MLP predicts candidate logits.
AudioANN-locfirst: a position head predicts 3D source position, and candidate logits are the negative distances between that position and each candidate bird.
AudioANN-hybrid: a learnable interpolation blends direct logits and localization-derived logits.
RadioANN family. RadioANN is the radio-only analogue. Its backbone is a Conv1d encoder over the 7 raw radio temporal channels per bird, with channel progression followed by global temporal pooling. Again we evaluate three variants:
RadioANN-direct: score each bird directly from its radio embedding.
RadioANN-locfirst: predict a per-bird 3D position from the radio embedding and rank candidates by distance.
RadioANN-hybrid: blend direct radio scores with localization-derived scores.
These models are intentionally simple and expose how much attribution signal is available from straightforward audio or radio encoders before adding stronger spatial inductive bias.
Vision-Only Candidate Baselines
PoseMotion. PoseMotion is the pose-only control. For each bird and frame we derive a compact motion vector from the tracked 3D keypoints: relative body coordinates together with frame-to-frame deltas. A 2-layer MLP projects each frame to a hidden representation, and a 2-layer bidirectional GRU pools the short temporal window into a per-candidate motion embedding. Candidate logits are then predicted from the motion embedding concatenated with the bird’s normalized 3D position and head orientation. Localization is candidate-conditioned: the predicted source position is the softmax-weighted average of candidate positions. This baseline measures whether short-window beak/head/backpack motion alone is sufficient for caller attribution.
VideoCandidate. VideoCandidate is the RGB active-speaker-style control. We extract synchronized top-view and back-view crop tubes around each visible bird from the composite video, using a crop budget and 4 sampled frames per event window. Each frame is encoded by a small CNN ( channels with strided convolutions), the two views are fused by an MLP, and a 2-layer bidirectional GRU pools the crop tube over time. Candidate logits are again produced from the fused visual embedding together with the bird’s normalized 3D position and head orientation. Localization is candidate-conditioned via the score-weighted candidate position. This row is the closest benchmark analogue to classical active-speaker baselines such as AVA-style, TalkNet-style, and UniTalk-style visual speaker attribution [51, 41] (Tao et al. 2021), but adapted to multi-bird candidate scoring rather than person-centric diarization.
Literature-Style SELD Adaptations
Shared spatial encoder. The ACCDOA, EINV2, and CST-former rows all begin from SALSA-Lite spatial audio features [42] and normalized candidate state. The convolutional variants use a shared spatial encoder composed of SALSA-Lite extraction followed by a CNN with channel progression ; the transformer variant uses a 3-layer SpatialConformer in the spirit of Conformer-style encoder blocks [24] with 4 attention heads and model width 192.
ACCDOA adaptation. Our ACCDOA adaptation follows the Multi-ACCDOA formulation [55]: it applies a 2-layer bidirectional GRU on top of the spatial encoder, predicts a framewise activity scalar, and regresses a framewise 3D position vector. The final source position is the activity-weighted average over frames. Attribution is then performed by converting distances between the predicted position and candidate bird positions into logits, optionally with an additional radio-conditioned position-bias scorer.
EINV2 adaptation. EINV2 retains the event-independent trackwise formulation [5]. A recurrent spatial encoder produces a pooled representation that predicts latent tracks, each with an activity logit and 3D position. Candidate attribution is obtained by matching candidate positions to the latent tracks through distance-based logits and taking the best track per candidate. This produces a free-localization model rather than a pure candidate-conditioned scorer.
CST-former adaptation. CST-former uses a transformer-style spatial encoder over SALSA-Lite features [57]. The pooled transformer representation drives both a 3D position head and a candidate scorer that consumes the audio embedding together with each bird’s position, head orientation, and optional radio context. It therefore sits between a free localization model and a candidate-conditioned scorer.
Physics-Guided Learned Spatial Baselines
SRP reranker. SRP reranker is the simplest learned extension of SRP. We first precompute frozen SRP-style features for every candidate: for each microphone pair we record the GCC-PHAT peak within a sample window around the expected TDOA, the narrow-window mean, and a peak-to-mean sharpness statistic. Concatenating these over 10 microphone pairs yields a 30-dimensional SRP descriptor per candidate. A small 3-layer MLP then scores each candidate from this descriptor plus candidate position, head orientation, and the 12-dimensional radio summary. The model therefore tests whether a small learned reranker can improve on physics without replacing it.
NeuralSRP. NeuralSRP keeps the same geometry-first philosophy but learns the spatial representation end to end. The input is a SALSA-Lite tensor composed of per-microphone log spectrograms and normalized inter-channel phase differences [42] (9 channels total). A ResNet-style encoder with squeeze-excitation blocks [28, 30] and frequency-aware normalization maps these features to a 256-dimensional audio embedding. Each candidate bird is represented by its expected TDOA vector relative to microphone 0, encoded by an MLP; optional radio temporal features are encoded by a small Conv1d stack. Attribution uses a learned per-candidate scorer over audio, TDOA, and optional radio embeddings, while localization comes from a separate MLP head on the shared audio embedding.
BeamFusion. BeamFusion adds explicit source-focused filtering. For each candidate bird, a learned beamformer computes physics steering vectors from the candidate 3D position and microphone geometry, refines them through an MLP, and applies them to the multichannel STFT to produce a beamformed per-bird spectrogram. These per-bird queries are fused with a global SALSA-Lite + SpatialConformer context stream through a transformer decoder, again using a Conformer-style spatial encoder [24]. Optional radio summaries are concatenated to the candidate queries before decoding. This is the most explicit multichannel “focus on each candidate position first, then classify” baseline in the suite.
Contrastive candidate model. Contrastive learns a shared embedding space for audio clips and candidate spatial signatures. The audio encoder is a multichannel mel CNN followed by a 2-layer bidirectional GRU and a projection to a 128-dimensional L2-normalized embedding. Each candidate is encoded from TDOA features across all microphone pairs, inverse-distance energy patterns to the microphones, normalized 3D position, and optional temporal radio embeddings. Attribution uses cosine similarity divided by a temperature parameter. This baseline is explicitly designed for transfer: it never learns a fixed slot-indexed class mapping.
Whisper candidate scorer. Whisper candidate uses a frozen Whisper encoder [48] as a pretrained audio backbone. Each microphone channel is resampled to 16 kHz, encoded independently by Whisper, pooled with a VocSim-style mean-over-time and mean-over-feature operation [3], and fused across microphones to a 256-dimensional audio embedding. Candidate features contain bird position, head orientation, and optional radio summary; a small MLP combines the audio embedding, candidate embedding, and their elementwise interaction to score candidates. This baseline measures whether a stronger pretrained audio representation materially changes transfer.
Withheld On-Body Reference
Withheld on-body reference. The withheld on-body reference uses only the per-bird accelerometer waveform provided by the validated on-body sensor. Each bird’s accelerometer trace is converted to a log-power spectrogram with and hop length 64. A shared 2D CNN with channel progression and max-pooling produces a per-bird feature map. Mean and max pooled features are concatenated and mapped through a small MLP to one score per bird. The predicted 3D position is the position of the predicted bird. This model is reported only as a ceiling for candidate-conditioned attribution and localization, not as a benchmark competitor.
Training Protocol (All Learned Models)
Unless otherwise stated, all learned baselines are trained with AdamW and a cosine-annealing schedule. Model-family-specific settings are summarized in Table 17; this section lists the conventions shared across families.
| Family | Architecture | LR | WD | Epochs | Batch | Seeds | Label smooth |
| SELDnet | CRNN of Adavanne et al. [ [1] | 60 | 64 | 3 | 0.1 | ||
| SRP reranker | ours | 40 | 128 | 5 | 0.05 | ||
| NeuralSRP | SALSA-Lite input Nguyen et al., [42] | 8 | 32 | 3 | 0.1 | ||
| PairRank | ours | 8 | 32 | 3 | 0.03 | ||
| SpatialScorer | ours | 8 | 16 | 3 | 0.1 | ||
| Contrastive | ours | 8 | 16 | 3 | 0.1 | ||
| Whisper candidate | frozen Whisper Radford et al., [48] | 6 | 8 | 3 | 0.0 | ||
| ACCDOA / EINV2 / CST-former | cited output heads, shared encoder | 8 | 16 | 3 | 0.0 | ||
| BeamFusion | ours | 8 | 16 | 3 | 0.1 | ||
| AudioANN / RadioANN | ours | 8 | 32 | 3 | 0.1 | ||
| PoseMotion | ours | 8 | 32 | 3 | 0.1 | ||
| VideoCandidate | ours | 8 | 16 | 3 | 0.1 | ||
| On-body ref. | ours | 6 | 32 | 3 | 0.0 |
Table 17. Shared hyperparameters across learned baselines. Architecture: source of the model design (ours = introduced in this work). Optimizer settings are shared defaults and were not tuned per method.
Optimizer and schedule. All learned baselines use AdamW with cosine annealing. SELDnet uses learning rate and weight decay . PairRank, SpatialScorer, the literature-style SELD baselines, AudioANN/RadioANN, BeamFusion, Contrastive, and Whisper-candidate use learning rate and weight decay in the range to . SRP reranker uses a larger learning rate of because it only trains a small MLP on frozen SRP descriptors. SELDnet is trained for 60 epochs on the session split and 40 epochs on LOO-style controls. Most matrix baselines are trained for shorter budgets because they are swept over many more cells: PairRank, SpatialScorer, ACCDOA, EINV2, CST-former, AudioANN/RadioANN, BeamFusion, and Contrastive use 8 epochs; Whisper-candidate uses 6 epochs; NeuralSRP uses 8 epochs; SRP reranker uses 40 epochs.
Loss. for models with an explicit localization head, where:
: Cross-entropy with label smoothing .
: Mean squared error between predicted and true vocalizer position (normalized coordinates).
Data augmentation. Applied with the following probabilities per batch:
Channel dropout (25%): 1–2 randomly selected microphone channels are scaled by .
Per-channel gain variation (30%): Each channel is multiplied by an independent gain factor .
Additive noise (40%): Gaussian noise with standard deviation is added to all channels.
Gradient clipping. Maximum gradient norm of 1.0.
Batch size. Batch size depends on model cost. SELDnet uses 64. PairRank and SRP reranker use 32–128. SpatialScorer and the literature-style SELD baselines use 16. AudioANN/RadioANN use 32. Whisper-candidate uses 8 because the frozen Whisper encoder dominates memory.
Seeds. The matrix-style transfer families (PairRank, SpatialScorer, ACCDOA, EINV2, CST-former, AudioANN/RadioANN, BeamFusion, Contrastive, and Whisper-candidate), together with SELDnet, NeuralSRP, and the withheld on-body reference, are reported as mean standard deviation over 3 seeds {42, 123, 456}. The SRP reranker uses 5 seeds {42, 123, 456, 789, 1337}; SRP 20, LoudestMic, and Random are deterministic and reported without seed variation.
Early stopping. Early stopping is not used. All models are trained for the full epoch budget and the best validation-accuracy checkpoint is selected.
Compute resources. All reported training runs were executed on a single local workstation with one NVIDIA RTX 3090 GPU (24 GB VRAM); most scripts used up to 4 CPU data-loader workers. Summing successful stage durations from the saved logs, the reported main benchmark matrix required approximately 42.0 GPU-hours end-to-end. The completed comparable diagnostics add 6.6 hours, and the two vision-only controls add 1.3 hours (PoseMotion) and 3.3 hours (VideoCandidate, including offline crop-cache precomputation), for a total of about 53.1 GPU-hours across the experiments reported in the paper. On a single 3090 this corresponds to roughly 2.2 wall-clock days for the retained experiments alone.
Runtime variation across families. Most retained matrix families fall in the 1–2 hour range when their three benchmark regimes are run end-to-end (e.g., PairRank 1.1 h, SpatialScorer 1.3 h, ACCDOA 1.1 h, EINV2 1.4 h, CST-former 1.6 h, Contrastive 1.1 h, Whisper candidate 2.3 h). The main outliers are the heavier LOO-sensitive or preprocessing-heavy families: benchmark-family ablations (5.4 h), AudioANN/RadioANN (5.4 h), SRP reranker (7.9 h), NeuralSRP (3.4 h), and BeamFusion (8.9 h). Across successful matrix stages, the median stage duration is 35.4 minutes, with a range from 14.9 minutes to 4.1 hours.
Learning curves. Figure 3 shows validation accuracy as a function of training epoch for the seven matrix-family baselines whose training logs contain per-epoch validation metrics, under the three held-out-experiment splits. Shaded bands are one standard deviation across the three training seeds. Most models are close to their final validation accuracy by the end of the 8-epoch budget used in the main sweep, but CST-former has not reached a plateau by epoch 8 on held-out BP05; Appendix C.12 tests the effect of the budget directly. SpatialScorer, PairRank, and the withheld on-body reference log only best-epoch summaries and are omitted from the per-epoch plot.
Figure 3. Validation accuracy per epoch for matrix-family baselines on the three held-out-experiment splits. Lines are seed means, shaded bands are standard deviation across the three training seeds.
Training Budget Control
To test whether the 8-epoch budget understated CST-former and Whisper-candidate, we reran both for 60 epochs with the same architecture, preprocessing, optimizer, splits, seeds, and validation-only checkpoint selection, and ran ACCDOA under the same extension as a control (Table 16).
| Method | Evaluation aggregate | Submitted | 60-epoch |
| CST-former | LOO | 15.0 | |
| CST-former | Held-out exp. mean | 30.0 | 33.7 |
| Whisper-candidate | LOO | 12.1 | |
| Whisper-candidate | Held-out exp. mean | 31.4 | 30.3 |
| ACCDOA | LOO | 55.4 | 56.7 |
| ACCDOA | Session-disjoint | 65.6 | 67.1 |
Table 16. Attribution accuracy (%) with the submitted budget and with 60 epochs. Held-out exp. mean averages BP01, BP02, and BP05.
CST-former’s selected epochs span 23–55 (median 44), and its LOO accuracy rises from 15.0% to , so its submitted result is budget-sensitive. Whisper-candidate does not improve, and ACCDOA changes by 1.3 points LOO and 1.5 points session-disjoint. Training budget therefore affected CST-former’s absolute score but did not create the formulation-level pattern.
Configuration choices. Configurations were fixed once and reused across all folds and seeds, with architectures following the cited formulations and optimizer settings taken from the shared defaults of Table 17; no setting was tuned per method. ACCDOA, EINV2, and CST-former keep their published output formulations on a shared spatial encoder (192-dimensional, 64 mel bands) rather than their published model sizes. Checkpoints are selected on validation accuracy only; no test result selected a configuration, budget, or checkpoint. Table 17 lists each family’s optimizer settings, budget, batch size, seeds, and label smoothing, and the loss weights are given below it.
Evaluation Protocol
Session-disjoint split. Train: 19,365 samples. Val: 6,869 samples. Test: 7,391 samples. By experiment, the session split allocates BP01 as 12,881/1,008/2,330, BP02 as 4,455/4,155/2,967, and BP05 as 2,029/1,706/2,094 for train/val/test.
Held-out experiment. We rotate the held-out experiment across BP01, BP02, and BP05. In each fold, training uses the remaining two experiments and testing uses the held-out one. The resulting train/val/test sizes are 6,484/5,861/16,219 for held-out BP01, 14,910/2,714/11,577 for held-out BP02, and 17,336/5,163/5,829 for held-out BP05. This is the main transfer regime in the paper because it combines identity shift with experiment-level nuisance changes such as year, cohort, and recording date.
Leave-one-bird-out (LOO). We run within-experiment LOO controls for all experiments. BP01 has 4 folds with train sizes 9,418–12,535, validation sizes 1,066–1,407, and test sizes 2,277–5,735. BP02 has 3 folds with train sizes 5,989–8,569, validation sizes 665–943, and test sizes 2,065–4,923. BP05 has 3 folds with train sizes 1,974–5,201, validation sizes 222–572, and test sizes 56–3,633. Same recording dates appear in both train and test; only the bird identity changes.
Metrics.
Top-1 accuracy: Fraction of correctly attributed vocalizations.
Macro-F1: Unweighted mean of per-class F1 scores.
Localization error: Euclidean error in 3D source position, reported separately for free localization and candidate-conditioned localization.
Shuffle intervention. For the modality-shuffle diagnostic, we run 10 independent random shuffles (permuting bird indices within each sample for the specified modality) and report mean std of test accuracy. Audio is never shuffled. Shuffling destroys the correspondence between a bird’s position/radio features and its identity while preserving the marginal distribution of feature values.
Training Hyperparameters
Table 17 summarizes training hyperparameters across all learned baselines.
All learned models use AdamW with cosine-annealing schedule and gradient clipping at max norm 1.0. Loss is unless otherwise noted, where varies by model (default 0.1; AudioANN/RadioANN use 0.5; Whisper candidate uses 0.05; SELD literature baselines use 0.05; SRP reranker and withheld on-body reference use pure cross-entropy).
Shared Sub-Modules
JointRadioTemporalEncoder. Used by ACCDOA (radio bias scorer), EINV2, and CST-former radio-context variants.
Input: raw radio signals for all birds.
Joint CNN: Three Conv2d layers over the birdtime grid: Conv2d BN ReLU Conv2d BN ReLU Conv2d BN ReLU. Mean-pool over time per-bird 64-dim.
Cross-bird attention: 1-layer Transformer encoder (4 heads, FFN dim 128, pre-norm). Learnable bird-position embeddings (up to 8 birds).
Outputs: Per-bird embedding: concatenated with global context Linear ReLU Linear. Global embedding: Linear ReLU Linear.
SALSALiteExtractor. Shared front-end for all SELD-literature baselines, NeuralSRP, and Beam-Fusion context stream.
Channel 0: log-mel spectrogram of the reference mic.
Channels 1–: normalized inter-channel phase differences (NIPDs) relative to mic 0.
Output: where and .
Descriptive and Comparative Statistics
This appendix reports descriptive and comparative statistics that support the interpretation of the benchmark results: detailed attribution analysis, intervention diagnostics, fold-level leave-one-bird-out performance, sample-weighted versus unweighted LOO summaries, localization results, and global transfer rankings across tasks.
Attribution: Detailed Analysis
This subsection expands the main-text ATTRIBUTION paragraph (§ 6).
Slot-collapse failure. SELDnet++ achieves the highest session score (86.5%) yet collapses to 0% LOO. The failure is diagnostic rather than accidental: although SELDnet++ includes a candidate-matching branch, training is dominated by its slot-indexed head, which can exploit stable within-experiment slot-to-bird mappings but cannot generalize when the held-out bird’s slot is absent during training. Removing that head in SELDnet++ open restores non-zero LOO (9.3%) at a large session cost (61.5%), confirming that the shortcut is both attractive and brittle; SELDnet (81.8%) and AudioANN-direct (72.1%) exhibit the same slot-collapse failure. SELDnet++ open is informative on its own: forced onto pure candidate matching with full candidate conditioning (3D position, head direction, radio), the SELDnet backbone lands at 61.5% session / 9.3% LOO, well below the geometry-first rows despite having access to the same candidate-side modalities, indicating that the architecture does not on its own convert candidate features into transferable attribution.
Regime-level results. Across regimes the common ingredient of the strongest rows is not a single backbone but a formulation: the highest observed values come from EINV2 on held-out BP01 and SpatialScorer on BP02 and BP05, and SRP, SpatialScorer, ACCDOA, EINV2, and AudioANN-locfirst show comparable LOO transfer, all of them scoring visible candidates directly or routing attribution through localization. These values are descriptive: the paired tests in Table 26 do not resolve the order of these methods. Uniform-fold and sample-weighted LOO give 55.4/60.8% for ACCDOA, 55.2/60.2% for EINV2, 53.0/57.5% for AudioANN-locfirst, and 53.5/52.9% for SRP (Table 22); removing any single bird keeps the AudioANN-locfirst mean between 50.8% and 56.6%, and excluding any complete experiment keeps it at or above 44.7% while AudioANN-direct stays at 0.0%. The BP05 pattern is not reducible to class imbalance alone.
| Method | Uniform / per-bird recall | Sample-weighted |
| Random | 27.0 | 27.9 |
| SRP | 53.5 | 52.9 |
| SELDnet | 0.0 | 0.0 |
| SpatialScorer | 49.5 | 58.3 |
| NeuralSRP | 19.2 | 22.8 |
| BeamFusion | 27.6 | 30.5 |
| Contrastive | 40.4 | 46.8 |
| Whisper candidate | 12.1 | 13.3 |
| ACCDOA | 55.4 | 60.8 |
| EINV2 | 55.2 | 60.2 |
| CST-former | 15.0 | 16.3 |
| PairRank | 6.7 | 6.8 |
| AudioANN-direct | 0.0 | 0.0 |
| AudioANN-locfirst | 53.0 | 57.5 |
| AudioANN-hybrid | 0.0 | 0.0 |
| RadioANN-direct | 19.7 | 20.1 |
| RadioANN-locfirst | 28.4 | 30.5 |
| RadioANN-hybrid | 23.6 | 25.4 |
| PoseMotion | 17.0 | 19.4 |
| VideoCandidate | 26.9 | 28.3 |
| On-body ref. | 91.3 | 90.9 |
Table 22. LOO accuracy / per-bird recall, uniform vs. sample-weighted across the 10 held-out-bird folds. Uniform columns match Table 3 and are also the per-bird recall average. Weighted columns use the test-set sizes in Table 11. Both columns average seeds within each fold before aggregating.
| Comparison | acc | ||
| SpatialScorer − SRP ±20 | -0.007 | 0.94604 | 1.00000 |
| ACCDOA − SRP ±20 | +0.022 | 0.45483 | 1.00000 |
| EINV2 − SRP ±20 | +0.022 | 0.37573 | 1.00000 |
| AudioANN-locfirst − SRP ±20 | -0.000 | 0.49731 | 1.00000 |
| CST-former − SRP ±20 | -0.343 | 0.00049 | 0.00391 |
| RadioANN-locfirst − SRP ±20 | -0.231 | 0.00464 | 0.03247 |
| ACCDOA − SpatialScorer | +0.029 | 0.63550 | 1.00000 |
| EINV2 − SpatialScorer | +0.030 | 0.63550 | 1.00000 |
Table 26. Selected paired transfer significance tests for attribution accuracy. Positive acc means that the row method improves mean fold-wise transfer accuracy over the column method.
Full attribution matrix. Table 18 gives the full attribution matrix (24 methods plus the withheld on-body reference) corresponding to the compact main-text Table 3.
| Method | Session | Held-out BP01 | Held-out BP02 | Held-out BP05 | LOO mean |
| Random | 28.3 ± 0.5 | 24.8 ± 0.2 | 33.3 ± 0.6 | 24.6 ± 0.3 | 27.0 ± 4.6 |
| LoudestMic | 31.1 | 18.9 | 39.5 | 26.3 | 27.3 ± 11.2 |
| SRP ±20 | 51.0 | 52.4 | 62.2 | 35.8 | 53.5 ± 27.5 |
| SELDnet | 81.8 ± 1.2 | 34.3 ± 0.4 | 38.6 ± 0.2 | 41.7 ± 8.6 | 0.0 ± 0.0 |
| SELDnet++ | 86.5 ± 0.4 | 39.8 ± 0.8 | 44.0 ± 1.2 | 36.9 ± 3.5 | 0.0 ± 0.0 |
| SELDnet++ open | 61.5 ± 0.4 | 34.6 ± 1.4 | 45.4 ± 0.0 | 29.2 ± 1.2 | 9.3 ± 4.8 |
| PairRank | 37.9 ± 1.7 | 32.7 ± 0.2 | 35.8 ± 1.1 | 29.7 ± 1.5 | 6.7 ± 5.0 |
| SpatialScorer | 73.0 ± 1.2 | 56.7 ± 1.6 | 69.1 ± 1.0 | 55.0 ± 1.2 | 49.5 ± 22.0 |
| SRP reranker | 45.1 ± 1.0 | 41.0 ± 1.2 | 51.6 ± 0.7 | 32.3 ± 0.5 | 38.5 ± 9.4 |
| NeuralSRP | 42.9 ± 0.9 | 26.0 ± 0.5 | 45.5 ± 0.8 | 49.2 ± 0.4 | 19.2 ± 13.8 |
| BeamFusion | 66.6 ± 2.5 | 44.9 ± 3.4 | 54.5 ± 3.3 | 34.1 ± 2.3 | 27.6 ± 12.2 |
| Contrastive | 72.7 ± 0.7 | 53.2 ± 1.2 | 66.4 ± 0.8 | 52.5 ± 1.9 | 40.4 ± 20.0 |
| Whisper candidate | 40.5 ± 0.8 | 27.6 ± 2.0 | 34.3 ± 0.2 | 32.2 ± 1.9 | 12.1 ± 6.0 |
| ACCDOA | 65.6 ± 0.2 | 58.1 ± 2.0 | 63.4 ± 1.3 | 38.6 ± 0.9 | 55.4 ± 21.8 |
| EINV2 | 65.8 ± 0.3 | 58.2 ± 1.9 | 66.6 ± 1.4 | 38.2 ± 0.3 | 55.2 ± 22.1 |
| CST-former | 49.8 ± 0.6 | 28.4 ± 0.6 | 35.8 ± 0.7 | 25.8 ± 0.9 | 15.0 ± 5.6 |
| AudioANN-direct | 72.1 ± 3.1 | 35.0 ± 0.4 | 42.4 ± 0.5 | 28.8 ± 12.0 | 0.0 ± 0.0 |
| AudioANN-locfirst | 61.8 ± 0.6 | 56.3 ± 1.1 | 63.0 ± 0.2 | 36.3 ± 0.5 | 53.0 ± 19.2 |
| AudioANN-hybrid | 68.0 ± 3.8 | 35.8 ± 1.3 | 39.5 ± 2.7 | 22.8 ± 10.7 | 0.0 ± 0.0 |
| RadioANN-direct | 28.5 ± 0.4 | 29.2 ± 0.9 | 39.7 ± 0.3 | 25.4 ± 0.5 | 19.7 ± 9.5 |
| RadioANN-locfirst | 37.6 ± 0.1 | 32.4 ± 0.6 | 38.8 ± 0.2 | 29.3 ± 0.2 | 28.4 ± 11.6 |
| RadioANN-hybrid | 33.7 ± 0.4 | 30.5 ± 0.6 | 41.5 ± 0.6 | 28.3 ± 0.9 | 23.6 ± 11.8 |
| PoseMotion | 42.5 ± 0.2 | 35.8 ± 0.7 | 43.4 ± 0.3 | 28.8 ± 0.7 | 17.0 ± 7.3 |
| VideoCandidate | 44.6 ± 0.8 | 25.0 ± 0.5 | 37.2 ± 0.1 | 33.3 ± 1.1 | 26.9 ± 10.3 |
| Ceiling / sanity-check row | |||||
| On-body ref. | 99.2 ± 0.1 | 91.9 ± 1.0 | 98.0 ± 0.6 | 96.6 ± 0.2 | 91.3 ± 7.6 |
Table 18. Full vocal attribution matrix across benchmark regimes. All methods solve the same task on the released events: given a synchronized multimodal window, pick the vocalizing bird from the set of candidates visible at that moment. Entries are raw attribution accuracy (%). Session tests new recordings of known birds. Held-out BP01/BP02/BP05 rotate the test experiment and measure transfer across experiments and individuals. LOO mean averages within-experiment leave-one-bird-out folds and isolates identity shift under a fixed sensing setup. Learned rows report mean ± standard deviation; deterministic rows are shown without ±. The withheld on-body reference is included only as a ceiling.
Modality Interventions: Detailed Analysis
This subsection expands the main-text intervention paragraph (§ 6).
CST-former and further interpretation. Under the short-window feature summary used in Table 3, radio is not intrinsically uninformative but is largely redundant once candidate geometry is known. CST-former shows small and sometimes counterintuitive deltas throughout, consistent with a model that does not strongly use any single candidate-side modality.
Transfer-rank significance. Table 24–Table 26 complete the picture: the strongest attribution rows form a compact spatially grounded tier under mean transfer rank, with SpatialScorer, SRP, ACCDOA, EINV2, and AudioANN-locfirst statistically indistinguishable, while CST-former and RadioANN-locfirst remain significantly weaker.
| Method | Acc. | 95% CI | Rank | Wins |
| Random | 27.2 | [24.9, 29.5] | 12.9 | 0 |
| SRP ±20 | 52.7 | [40.0, 65.5] | 5.9 | 3 |
| SELDnet | 8.8 | [0.0, 18.0] | 20.2 | 0 |
| PairRank | 12.7 | [6.7, 19.6] | 16.4 | 0 |
| SpatialScorer | 52.0 | [40.7, 61.2] | 4.1 | 5 |
| SRP reranker | 39.2 | [34.4, 43.8] | 6.9 | 0 |
| NeuralSRP | 24.1 | [15.7, 32.2] | 11.7 | 0 |
| BeamFusion | 31.5 | [24.4, 38.3] | 9.6 | 0 |
| Contrastive | 44.4 | [33.7, 53.7] | 6.4 | 0 |
| Whisper cand. | 16.5 | [11.4, 22.0] | 15.9 | 0 |
| ACCDOA | 54.9 | [44.2, 64.5] | 3.6 | 1 |
| EINV2 | 55.0 | [43.9, 64.6] | 3.8 | 3 |
| CST-former | 18.5 | [14.2, 23.0] | 15.2 | 0 |
| AudioANN-dir | 8.2 | [0.0, 17.3] | 18.1 | 0 |
| AudioANN-loc | 52.7 | [42.9, 61.5] | 4.7 | 1 |
| AudioANN-hyb | 7.5 | [0.0, 16.1] | 18.7 | 0 |
| RadioANN-dir | 22.4 | [17.1, 27.6] | 13.1 | 0 |
| RadioANN-loc | 29.6 | [24.1, 35.0] | 10.2 | 0 |
| RadioANN-hyb | 25.9 | [19.8, 31.7] | 11.8 | 0 |
| PoseMotion | 21.4 | [15.8, 27.5] | 13.5 | 0 |
| VideoCandidate | 28.1 | [22.8, 33.2] | 12.4 | 0 |
| On-body ref. | 92.7 | [88.4, 96.1] | – | – |
Table 24. Global transfer attribution summary. Mean accuracy over 13 transfer tasks (3 held-out + 10 LOO). Rank and wins computed task-wise.
Position-noise robustness. A test-time perturbation of candidate 3D position with Gaussian noise at cm (Table 27) confirms the same split: SpatialScorer, Contrastive, BeamFusion, ACCDOA, and EINV2 degrade gracefully with noise magnitude, while NeuralSRP and SELDnet++ open are effectively flat under position noise, meaning they do not route attribution through candidate geometry. Head-orientation perturbation moved every cell by less than 0.1 point and so does not contribute to attribution in any of these rows.
| Regime | Method | Clean | cm | cm | cm | |
| Session | SpatialScorer | 71.8 | 71.5 | 68.3 | 62.1 | -9.7 |
| Session | Contrastive | 72.0 | 71.9 | 69.9 | 64.7 | -7.3 |
| Session | BeamFusion | 67.9 | 67.3 | 63.0 | 55.1 | -12.8 |
| Session | ACCDOA | 64.5 | 64.4 | 61.9 | 56.2 | -8.3 |
| Session | EINV2 | 65.2 | 65.1 | 62.6 | 56.5 | -8.7 |
| Session | NeuralSRP | 43.8 | 43.8 | 43.7 | 43.6 | -0.2 |
| Session | SELDnet++ open | 47.7 | 47.7 | 47.6 | 47.6 | -0.1 |
| Heldout (avg. BP01/02/05) | SpatialScorer | 54.9 | 54.8 | 53.0 | 49.3 | -5.6 |
| Heldout (avg. BP01/02/05) | Contrastive | 59.8 | 59.8 | 58.3 | 54.1 | -5.7 |
| Heldout (avg. BP01/02/05) | BeamFusion | 45.5 | 45.3 | 42.4 | 38.1 | -7.4 |
| Heldout (avg. BP01/02/05) | ACCDOA | 52.1 | 51.9 | 49.9 | 45.5 | -6.6 |
| Heldout (avg. BP01/02/05) | EINV2 | 52.2 | 52.0 | 50.1 | 45.5 | -6.7 |
| Heldout (avg. BP01/02/05) | NeuralSRP | 40.1 | 40.1 | 40.1 | 40.1 | +0.0 |
| Heldout (avg. BP01/02/05) | SELDnet++ open | 36.3 | 36.3 | 36.3 | 36.3 | +0.0 |
| LOO (avg. 10 birds) | SpatialScorer | 47.0 | 46.9 | 44.4 | 40.3 | -6.7 |
| LOO (avg. 10 birds) | Contrastive | 41.6 | 41.6 | 40.4 | 37.3 | -4.3 |
| LOO (avg. 10 birds) | BeamFusion | 28.6 | 28.3 | 26.0 | 22.8 | -5.8 |
| LOO (avg. 10 birds) | ACCDOA | 54.4 | 54.4 | 52.3 | 47.6 | -6.8 |
| LOO (avg. 10 birds) | EINV2 | 54.4 | 54.3 | 52.6 | 47.5 | -6.9 |
| LOO (avg. 10 birds) | NeuralSRP | 18.5 | 18.5 | 18.5 | 18.5 | +0.0 |
| LOO (avg. 10 birds) | SELDnet++ open | 10.3 | 10.3 | 10.3 | 10.3 | +0.1 |
Table 27. Mean attribution accuracy under test-time perturbation of candidate 3D position. Rows are grouped by evaluation regime. is the drop at cm relative to the clean column. Orientation perturbations (not shown) moved every cell by < 0.1 pt.
Full intervention table. Table 19 gives the full modality-intervention matrix across all four evaluation splits, with accuracy / macro-F1 / localization-error entries; the main-text Table 3 shows only the LOO-mean accuracy deltas, including the trivially-zero deltas of the slot-collapse rows (SELDnet++, AudioANN-direct) for completeness.
| Split | Condition | ACCDOA | EINV2 | SpatialScorer | CST-former |
| Held-out BP01 | Base | 58.4/56.7/23.2 | 57.9/56.1/24.2 | 58.5/57.1/17.0 | 28.4/26.6/22.0 |
| Held-out BP01 | zero audio | -27.9/-27.2/+9.2 | -24.5/-23.8/+3.8 | -35.9/-35.4/+15.3 | +4.1/+4.2/+5.9 |
| Held-out BP01 | shuffle geometry | -33.7/-32.5/+0.0 | -33.1/-31.9/+0.0 | -33.5/-32.5/+0.0 | -0.7/-0.7/+0.0 |
| Held-out BP01 | zero radio | +0.0/+0.0/+0.0 | +0.0/+0.0/+0.0 | +1.2/+1.4/+0.0 | +5.1/+5.8/+0.0 |
| Held-out BP02 | Base | 61.2/59.4/22.7 | 66.6/65.2/21.8 | 66.3/65.7/19.8 | 36.3/35.3/25.8 |
| Held-out BP02 | zero audio | -25.2/-24.3/+10.2 | -29.8/-29.7/+11.5 | -27.1/-28.8/+14.2 | -2.7/-2.3/+3.8 |
| Held-out BP02 | shuffle geometry | -27.5/-26.8/+0.0 | -32.7/-32.5/+0.0 | -31.9/-32.5/+0.0 | -1.3/-1.4/+0.0 |
| Held-out BP02 | zero radio | +0.0/+0.0/+0.0 | +0.0/+0.0/+0.0 | -0.8/-1.1/+0.0 | -0.1/+0.3/+0.0 |
| Held-out BP05 | Base | 37.8/26.9/26.9 | 38.1/27.2/26.7 | 53.4/32.3/23.9 | 25.7/17.9/26.2 |
| Held-out BP05 | zero audio | -9.1/-7.1/+5.0 | -6.6/-6.2/+2.9 | -3.2/-5.7/+8.0 | -1.3/+0.3/+3.7 |
| Held-out BP05 | shuffle geometry | -11.9/-9.5/+0.0 | -12.3/-10.1/+0.0 | -16.8/-10.5/+0.0 | -3.9/-2.7/+0.0 |
| Held-out BP05 | zero radio | +0.0/+0.0/+0.0 | +0.0/+0.0/+0.0 | -14.8/-5.1/+0.0 | +6.0/+3.0/+0.0 |
| LOO mean | Base | 56.1/19.4/23.7 | 54.4/18.9/25.0 | 48.7/17.2/19.8 | 15.1/7.1/23.0 |
| LOO mean | zero audio | -27.4/-7.3/+8.2 | -26.4/-7.0/+6.9 | -26.2/-7.9/+11.0 | -2.3/-1.3/+3.1 |
| LOO mean | shuffle geometry | -29.0/-7.6/+0.0 | -26.9/-6.8/+0.0 | -25.2/-7.1/+0.0 | -0.2/+0.0/+0.0 |
| LOO mean | zero radio | +0.0/+0.0/+0.0 | +0.0/+0.0/+0.0 | +6.3/+1.7/+0.0 | +10.8/+4.1/+0.0 |
Table 19. Full modality interventions. Base rows report accuracy / legacy fold-wise macro-F1 / localization error in cm; delta rows report changes from base in points / points / cm. The macro-F1 entries are retained only as intervention diagnostics within the same split and should not be read as the LOO class-balanced benchmark summary. Positive localization deltas indicate worse error. Interventions are evaluated on a single seed-42 checkpoint, so base accuracies may differ slightly from the multi-seed means reported in Table 18 while remaining within the reported seed variance.
Per-Bird Leave-One-Bird-Out Results
Table 20 reports fold-level LOO attribution accuracy for representative methods. Fold difficulty is heterogeneous and not reducible to class frequency alone: the smallest BP05 bird (56 test events) is not uniformly the hardest fold, while the largest BP05 fold can still break otherwise strong models. The withheld on-body reference remains strong on all BP05 folds, which supports the interpretation that difficult folds reflect modeling challenges rather than invalid labels.
| Bird | Test n | SRP 20 | SpatialScorer | ACCDOA | EINV2 | Contrastive | VideoCandidate | On-body ref. | AudioANN-locfirst |
| BP01-0 | 3963 | 75.7 | 61.9 3.3 | 67.4 0.5 | 67.8 0.7 | 46.7 1.3 | 30.6 0.4 | 98.6 0.2 | 63.3 0.4 |
| BP01-1 | 2277 | 31.8 | 60.9 2.6 | 65.0 1.6 | 66.3 0.8 | 58.7 2.7 | 46.9 0.1 | 93.8 0.8 | 56.8 0.8 |
| BP01-2 | 4244 | 71.2 | 62.9 0.6 | 73.5 0.4 | 73.9 0.8 | 63.4 1.3 | 26.5 0.2 | 86.8 2.5 | 72.5 0.3 |
| BP01-3 | 5735 | 30.5 | 71.1 1.2 | 68.6 1.2 | 69.1 0.8 | 55.1 3.2 | 38.2 1.0 | 98.1 0.5 | 69.3 1.1 |
| BP02-0 | 2065 | 88.8 | 40.7 1.1 | 71.1 1.0 | 71.0 1.6 | 35.8 1.2 | 14.2 0.3 | 97.2 0.4 | 62.8 1.7 |
| BP02-1 | 4589 | 61.0 | 63.8 1.1 | 62.3 0.7 | 60.7 0.8 | 28.9 0.5 | 27.8 1.5 | 79.1 9.5 | 59.6 2.1 |
| BP02-2 | 4923 | 54.1 | 64.3 1.7 | 73.0 0.9 | 71.4 3.5 | 59.1 3.2 | 30.3 0.9 | 97.1 0.4 | 65.5 4.3 |
| BP05-0 | 2140 | 85.6 | 30.9 2.0 | 25.4 2.7 | 22.8 2.9 | 23.7 0.9 | 11.7 0.7 | 90.4 2.6 | 21.5 0.4 |
| BP05-2 | 56 | 32.1 | 0.0 0.0 | 26.8 3.1 | 30.4 1.8 | 0.0 0.0 | 26.8 4.7 | 94.0 2.7 | 38.1 2.2 |
| BP05-3 | 3633 | 6.6 | 39.0 1.4 | 20.5 0.4 | 18.5 0.9 | 33.0 2.0 | 16.4 0.7 | 78.2 2.3 | 20.6 0.8 |
Table 20. Per-bird leave-one-bird-out accuracy (%). Birds are rows and representative completed methods are columns. Test n is the number of held-out test events in that fold. For learned methods, entries report mean standard deviation across seeds within that held-out bird fold.
What predicts fold difficulty? To separate sample-count and geometry effects, Table 21 correlates per-fold LOO accuracy across the 10 held-out bird folds with (i) test-set size and (ii) the held-out bird’s mean nearest-neighbor distance, derived from Table 12. The picture is mixed rather than one-factor. For learned candidate models, fold size is often the stronger coarse predictor, but it is not sufficient: BP05-0 and BP05-3 remain difficult despite 2,140 and 3,633 test events, and the on-body reference stays high on both. For SRP, nearest-neighbor separation is the clearer predictor, consistent with its direct dependence on acoustic ambiguity. These summaries support the main interpretation in the paper: BP05 imbalance increases variance, but the transfer ordering cannot be reduced to sample count alone.
| Method | ||
| SRP | -0.30 | 0.45 |
| SpatialScorer | 0.94 | -0.23 |
| ACCDOA | 0.44 | -0.11 |
| EINV2 | 0.44 | -0.11 |
| AudioANN-locfirst | 0.65 | -0.27 |
| On-body ref. | -0.06 | -0.20 |
Table 21. Simple predictors of LOO fold difficulty. Spearman correlation across the 10 held-out-bird folds between fold-wise LOO accuracy, test-set size, and the held-out bird’s mean nearest-neighbor distance.
Sample-Weighted LOO and Per-Bird Recall
The main-text LOO column averages the 10 held-out-bird folds uniformly, giving equal weight to each held-out vocalizer. Because each LOO test fold contains positive calls from exactly one physical bird, this uniform mean is the per-bird recall average across the 10 released individuals. This is the cleaner class-balanced summary than fold-wise macro-F1 over local slot indices, which assigns zero F1 to absent classes and can make the near-perfect on-body reference appear weak. To show that the benchmark conclusions do not depend on uniform fold weighting, Table 22 also reports sample-weighted LOO averages, where folds are weighted by their test-set size (Table 11). The sample-weighted column explicitly down-weights the very small BP05 Bird 2 fold (56 test events), and the same spatially grounded top tier remains visible.
Localization Results
Table 23 reports the full localization companion results referenced briefly in § 6. Free 3D localization and candidate-conditioned localization are kept separate because they correspond to different formulations of the task; the withheld on-body reference is meaningful only as a ceiling for the candidate-conditioned setting.
| Method | Session | Held-out BP01 | Held-out BP02 | Held-out BP05 | LOO mean |
| Free 3D localization | |||||
| ACCDOA | |||||
| EINV2 | |||||
| CST-former | |||||
| AudioANN-locfirst | |||||
| RadioANN-locfirst | |||||
| Candidate-conditioned localization | |||||
| Random candidate | |||||
| SELDnet | |||||
| PairRank | |||||
| SpatialScorer | |||||
| NeuralSRP | |||||
| BeamFusion | |||||
| Contrastive | |||||
| Whisper candidate | |||||
| PoseMotion | |||||
| VideoCandidate | |||||
| Candidate-conditioned ceiling | |||||
| On-body ref. |
Table 23. Localization results (mean error in cm, lower is better). Free 3D localization and candidate-conditioned localization are reported separately. The withheld on-body reference is a ceiling only for the candidate-conditioned setting.
Global Transfer Summaries
Following Demšar [ [10], we summarize the benchmark not only by aggregated regime means, but also by task-wise ranks over the 13 transfer tasks: the three held-out-experiment folds and the ten leave-one-bird-out folds. Table 24 reports the mean transfer accuracy for every reported attribution method, a percentile-bootstrap 95% interval for that mean obtained by resampling transfer tasks with replacement [12], the average task rank (lower is better), and the number of fold wins. The on-body reference is shown separately as a ceiling and is not included in the rank competition. Table 25 gives the same summaries for localization, with free and candidate-conditioned formulations kept separate.
| Method | Error | 95% CI | Rank | Wins |
| Free 3D localization | ||||
| ACCDOA | 25.1 | [21.8, 28.8] | 3.0 | 0 |
| EINV2 | 24.9 | [20.8, 29.6] | 2.5 | 4 |
| CST-former | 23.7 | [21.3, 26.0] | 2.3 | 6 |
| AudioANN-loc | 25.0 | [21.4, 28.9] | 2.7 | 3 |
| RadioANN-loc | 36.5 | [32.2, 41.0] | 4.5 | 0 |
| Candidate-conditioned | ||||
| SELDnet | 26.6 | [24.6, 28.4] | 7.5 | 0 |
| PairRank | 30.6 | [27.0, 34.5] | 8.4 | 0 |
| SpatialScorer | 19.9 | [17.4, 22.6] | 2.3 | 6 |
| NeuralSRP | 25.8 | [23.7, 28.4] | 5.7 | 1 |
| BeamFusion | 22.5 | [19.3, 25.9] | 3.3 | 4 |
| Contrastive | 22.9 | [19.9, 26.1] | 4.5 | 0 |
| Whisper cand. | 25.8 | [23.5, 28.5] | 6.0 | 0 |
| PoseMotion | 24.5 | [22.5, 27.2] | 4.6 | 1 |
| VideoCandidate | 22.9 | [20.8, 25.3] | 3.9 | 1 |
| On-body ref. | 2.8 | [1.4, 4.5] | – | – |
Table 25. Global transfer localization summary. Mean error (cm) over the same 13 tasks. Rank within formulation only.
Selected Paired Transfer Tests
For focused comparative claims among the strongest transfer methods, Table 26 reports paired two-sided Wilcoxon signed-rank tests across the same 13 transfer tasks. We additionally report Holm-adjusted values [29] for the family of accuracy comparisons in the table.
Geometry robustness under test-time perturbation
To measure how much the geometry-consuming rows rely on clean candidate state, we perturb candidate 3D positions with Gaussian noise at cm and head orientations by axis-angle rotations with Gaussian angle at test time only, applied to the best-validation checkpoint of each model. Table 27 reports mean attribution accuracy across seeds (and LOO folds, for the LOO row). Session is averaged over 3 seeds, Heldout over 9 (3 experiments × 3 seeds), and LOO over 30 (10 birds × 3 seeds). Orientation perturbation did not change attribution accuracy by more than 0.1 points on any model and split, so head direction is not contributing to attribution in this baseline suite under short windows; we omit orientation columns from the table for brevity. Under position noise, SpatialScorer, Contrastive, BeamFusion, ACCDOA, and EINV2 degrade gracefully (a few points at cm, up to points at cm on session, less on transfer splits), NeuralSRP is insensitive to position noise after training, and SELDnet++ open (trained at the reduced 8-epoch budget used for this sweep) is also effectively flat under position perturbation, consistent with a model that does not route attribution through candidate geometry.
Attribution and Inter-Candidate Distance
We stratified the three-seed LOO evaluation, pooling test events across all ten held-out-bird folds, by the distance from the vocalizer to its nearest competing candidate (Table 28). Accuracy increases as the nearest competitor moves farther away: ACCDOA rises by 26.4 points from the closest to the farthest bin, while random attribution changes by 4.8 points. We fit a logistic slope of correctness against standardized nearest-competitor distance separately for each seed and fold, average the seed-level slopes within each fold, and apply a two-sided Wilcoxon signed-rank test to the ten fold-level slopes. Eight of ten ACCDOA slopes are positive (); Holm correction across AudioANN-locfirst, ACCDOA, SpatialScorer, and SRP gives .
| Nearest competitor | ACCDOA | Random | Withheld on-body reference |
| < 10 cm | 46.6 | 26.3 | 91.2 |
| 10–20 cm | 54.2 | 26.7 | 91.1 |
| 20–30 cm | 61.0 | 27.8 | 91.2 |
| 30–50 cm | 67.3 | 28.9 | 90.8 |
| > 50 cm | 73.0 | 31.1 | 90.2 |
Table 28. LOO attribution accuracy (%) by distance from the vocalizer to its nearest competing candidate.
The array geometry continues to distinguish close candidates. Only 122 of 33,625 events (0.36%) have a competitor whose expected delays are within one audio sample of the vocalizer on every microphone pair; all 122 occur below 10 cm and represent 3.66% of that subset. A fixed GCC-PHAT decoder, which compares the observed microphone-pair correlations with the delays predicted from each candidate position and uses only the five-channel waveform, microphone geometry, and candidate positions, reaches 36.6% below 10 cm against 26.3% random attribution, and 38.4% overall against 27.9%; it remains above chance in every separation bin. Adding isotropic Gaussian noise to the verified vocalizer position and selecting the nearest candidate gives 91.0–98.9% attribution across the ten LOO folds at mm, 74.9–91.7% at mm, and 62.6–83.0% at mm.
Microphone Ablations
In a matched single-checkpoint-per-fold LOO ablation, SpatialScorer falls from 51.2% with five microphones to 45.2%, 38.5%, and 26.0% with four, three, and zero microphones. The 51.2% value is the intact baseline under this ablation protocol; Table 3 reports the separate three-seed LOO summary of 49.5 22.0%. On session-disjoint evaluation, permuting the channel-to-microphone correspondence reduces ACCDOA from 64.3% to 38.8%. Across the ten LOO folds, classical SRP ranges from 88.8% on BP02-0 to 6.6% on BP05-3, showing sensitivity to room geometry and reverberation.
Bioacoustic Front Ends and Clustering
We replaced the frozen mel front end of one candidate-scoring head with AVES, BirdNET, and animal2vec, fixing the head, protocol, 60-epoch budget, and three seeds (Table 29). All front ends reach 74–75% for familiar birds and 49–50% held-out, but 22.8–24.1% LOO against 27.0% for random attribution. During LOO training the held-out bird can appear as a silent candidate but never as the vocalizer, which permits a negative prior; held-out-experiment birds are absent in both roles. Stronger frozen acoustic representations preserve familiar-bird information but do not by themselves establish which tracked candidate produced the vocalization.
| Front end | Session | Held-out exp. mean | LOO |
| Random candidate | 28.3 | 27.6 | 27.0 |
| Mel spectrogram | 75.2 0.5 | 49.1 0.5 | 23.8 0.2 |
| AVES (self-supervised) | 75.4 0.7 | 49.3 1.0 | 24.1 0.5 |
| BirdNET (supervised) | 75.3 0.8 | 50.0 0.4 | 22.8 0.5 |
| animal2vec (self-supervised) | 74.3 0.5 | 50.0 1.4 | 22.8 0.6 |
Table 29. Frozen audio front ends under a fixed candidate-scoring head (accuracy, %). The held-out column averages BP01, BP02, and BP05.
For the unsupervised alternative, we clustered segmented events with -means given the true number of birds, using a frozen encoder trained on the same cohort. NMI and ARI reach 0.61 and 0.58, but assigning clusters to tracked birds still requires resolving among possible mappings.
Orientation-Weighted LoudestMic
For each bird we compute the cosine alignment between its head-to-beak direction and the direction to the loudest microphone, multiply the bird-to-microphone distance by , and select the minimum; the weight was fixed at 0.5 without tuning. The orientation-weighted variant changes accuracy by less than 0.6 points, in directions that vary across regimes (session-disjoint 31.1% 30.8%, held-out experiment 28.2% 28.7%, LOO 27.3% 27.9%), consistent with the small effect of orientation perturbation in Appendix D.8. We retain the original LoudestMic as the simpler energy-plus-distance reference.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The abstract, introduction, and experiments sections consistently frame the paper as a multimodal benchmark for spatial vocal attribution with transfer-oriented evaluation, withheld on-body reference rows, and modality interventions; the main claims are scoped to that benchmark setting.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 7 discusses the benchmark scope explicitly, including the single sensing installation, current scale, and restriction to verified single-vocalizer events.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [N/A]
Justification: The paper does not present theoretical results requiring formal assumptions or proofs.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The introduction and the Ethics, Licensing, and Release section provide public dataset and code URLs, and the appendix documents the preprocessing and training setup used for the benchmark.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: Sections 4–6 define the evaluation setup, and the appendix gives model-family-specific architectural and training details, including split construction and the preprocessing pipeline.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The main result tables report mean ± standard deviation for learned rows, Section 6 defines what those summaries capture in each regime, and Appendix Table 26 reports paired Wilcoxon signed-rank tests across the 13 transfer folds for the comparisons that support the main transfer claims.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes] Justification: Appendix § 13.11 reports the hardware used, data-loader configuration, aggregate runtime for the reported experiments, and the observed runtime range across model families.
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: The paper includes an Ethics, Licensing, and Release section, states the animal-protocol approval, and describes the release scope and licensing of the dataset and code.
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The Ethics, Licensing, and Release section states the intended research use of the benchmark, explains that no material societal risks are currently foreseen, and notes only a remote surveillance-style misuse scenario for related attribution or localization systems.
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: The release is an animal-behavior benchmark with no personally identifiable information: birds are referenced only by experiment-local backpack color codes, no human subjects are recorded, and no owner or facility metadata is shipped. All data were collected under Cantonal Veterinary Office of Zurich license ZH148/2020 (see Ethics, Licensing, and Release), the dataset is distributed under CC-BY-4.0, and the release scope (curated verified single-vocalizer events plus the V33DA++ extension) excludes the raw continuous recordings that would be needed for surveillance-style re-use.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper credits the external software and pretrained components it uses and states in the Ethics, Licensing, and Release section that these assets remain subject to their original licenses and terms of use.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper releases a new dataset and codebase, describes their scope and limitations in the main text, and points to the public release locations and appendix documentation.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: The work does not involve crowdsourcing or research with human subjects. The manual cross-modal validation of candidate events described in § 3 was performed by the first author as part of the research contribution, not by external annotators, so no participant instructions or compensation apply.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: The research does not involve human subjects.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: LLMs are not a component of the benchmark construction, models, or experimental methodology reported in the paper; any LLM assistance in prose editing did not contribute to dataset design, model design, experimental protocol, or result interpretation.
References
- [1]Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. CoRR, abs/1807.00129, 2018. URL http://arxiv.org/abs/1807.00129.
- [2]Victor N Anisimov, Joshua A Herbst, Andrei N Abramchuk, Alexander V Latanov, Richard H R Hahnloser, and Alexei L Vyssotski. Reconstruction of vocal interactions in a group of small songbirds. Nature Methods, 11(11):1135–1137, 2014. ISSN 1548-7105. doi: 10.1038/nmeth.3114. URL http://dx.doi.org/10.1038/nmeth.3114.
- [3]Maris Basha, Anja Zai, Sabine Stoll, and Richard Hahnloser. Vocsm: A training-free benchmark for zero-shot content identity in single-source audio, 2025. URL https://arxiv.org/abs/2512.10120.
- [4]Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkар, and Kris Kitani. Observation-centric sort: Rethinking sort for robust multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9686–9696, 2023.arxiv.org/abs/2203.14360
- [5]Yin Cao, Turab Iqbal, Qiuqiang Kong, Fengyan An, Wenwu Wang, and Mark D. Plumbley. An improved event-independent network for polyphonic sound event localization and detection. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 885–889, 2021. doi: 10.1109/ICASSP39728.2021.9413473.DOI
- [6]Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. The ami meeting corpus: A pre-announcement. In Machine Learning for Multimodal Interaction, page 28–39. Springer Berlin Heidelberg, 2006. ISBN 9783540255056. doi: 10.1007/11677482–3. URL http://dx.doi.org/10.1007/11677482_3.
- [7]Aneesh Chauhan, Hugo Loning, Evelien ter Avest, and Bhavya Botta. Birdaves in the wild: individual recognition as a step toward zebra finch communication networks, November 2025. URL http://dx.doi.org/10.1101/2025.11.20.689459.
- [8]Joon Son Chung, Jaesung Huh, Arsha Nagrani, Triantafyllos Afouras, and Andrew Zisserman. Spot the conversation: Speaker diarisation in the wild. In Interspeech 2020, page 299–303. ISCA, October 2020. doi: 10.21437/interspeech.2020-2337. URL http://dx.doi.org/10.21437/Interspeech.2020-2337.
- [10]Janez Demšar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7: 1–30, December 2006. ISSN 1532-4435.
- [11]Joseph Hubert DiBiase. A High-Accuracy, Low-Latency Technique for Talker Localization in Reverberant Environments Using Microphone Arrays. PhD thesis, Brown University, 2000.
- [12]Bradley Efron and R.J. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall/CRC, May 1994. ISBN 9780429246593. doi: 10.1201/9780429246593. URL http://dx.doi.org/10.1201/9780429246593.
- [13]Elena Eisenring, Marcel Eens, Jean-Nicolas Pradervand, Alain Jacot, Jan Baert, Eddy Uleanaers, Michiel Lathouwers, and Ruben Evens. Quantifying song behavior in a free-living, light-weight, mobile bird using accelerometers. Ecology and Evolution, 12(1), January 2022. ISSN 2045-7758. doi: 10.1002/ece3.8446. URL http://dx.doi.org/10.1002/ece3.8446.
- [14]Julie E. Elie and Frédéric E. Theunissen. The vocal repertoire of the domesticated zebra finch: a data-driven approach to decipher the information-bearing acoustic features of communication signals. Animal Cognition, 19(2):285–315, November 2015. ISSN 1435-9456. doi: 10.1007/s10071-015-0933-6. URL http://dx.doi.org/10.1007/s10071-015-0933-6.
- [15]Julie E. Elie and Frédéric E. Theunissen. Zebra finches identify individuals using vocal signatures unique to each call type. Nature Communications, 9(1), October 2018. ISSN 2041-1723. doi: 10.1038/s41467-018-06394-9. URL http://dx.doi.org/10.1038/s41467-018-06394-9.
- [16]Christine Evers, Heinrich W. Löllmann, Heinrich Mellmann, Alexander Schmidt, Hendrik Barfuss, Patrick A. Naylor, and Walter Kellermann. The locata challenge: Acoustic source localization and tracking. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 28:1620–1643, June 2020. ISSN 2329-9290. doi: 10.1109/TASLP.2020.2990485. URL https://doi.org/10.1109/TASLP.2020.2990485.
- [17]Valentin Gabeff, Haozhe Qi, Brendan Flaherty, Gencer Sümbül, Alexander Mathis, and Devis Tuia. Mammals: A multi-view video behavior monitoring dataset of wild mammals in the swiss alps. arXiv, 2025. doi: 10.48550/arXiv.2503.18223.DOI
- [18]Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res., 17(1):2096–2030, January 2016. ISSN 1532-4435.arxiv.org/abs/1505.07818
- [19]Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021, 2021. URL https://arxiv.org/abs/2107.08430.
- [20]Lisa F. Gill, Pietro B. D’Amelio, Nicolas M. Andreani, Hannes Sagunsky, Manfred C. Gahr, and Andries ter Maat. A minimum-impact, flexible tool to study vocal communication of small animals with precise individual-level resolution. Methods in Ecology and Evolution, 7(11):1349–1358, 2016. ISSN 2041-210X. doi: 10.1111/2041-210x.12610. URL http://dx.doi.org/10.1111/2041-210x.12610.
- [21]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Shlok Samtani, Kiran Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erappali, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abraham Gebreselassie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jáchym Kolář, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhengqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18995–19012, June 2022.arxiv.org/abs/2110.07058
- [22]Nianlong Gu, Kanghwi Lee, Maris Basha, Sumit Kumar Ram, Guanghao You, and Richard H. R. Hahnloser. Positive transfer of the whisper speech transformer to human and animal voice activity detection. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7505–7509, 2024. doi: 10.1109/ICASSP48485.2024.10447620.DOI
- [23]Eric Guizzo, Christian Marinoni, Marco Pennese, Xinlei Ren, Xiguang Zheng, Chen Zhang, Bruno Masiero, Aurelio Uncini, and Danilo Comminiello. L3das22 challenge: Learning 3d audio sources in a real office environment. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9186–9190, 2022. doi: 10.1109/ICASSP43922.2022.9746872.DOI
- [24]Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented transformer for speech recognition. In Helen Meng, Bo Xu, and Thomas Fang Zheng, editors, INTERSPEECH, pages 5036–5040. ISCA, 2020. URL http://dblp.uni-trier.de/db/conf/interspeech/interspeech2020.html#GulatiQCPZYHWZW20.arxiv.org/abs/2005.08100
- [25]Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization, 2020. URL https://arxiv.org/abs/2007.01434.
- [26]Masato Hagiwara, Benjamin Hoffman, Jen-Yu Liu, Maddie Cusimano, Felix Effenberger, and Katie Zacarian. Beans: The benchmark of animal sounds, 2022. URL https://arxiv.org/abs/2210.12300.
- [27]Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, USA, 2 edition, 2003. ISBN 0521540518.DOI
- [28]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.arxiv.org/abs/1512.03385
- [29]Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. ISSN 03036898, 14679469. URL http://www.jstor.org/stable/4615733.DOI
- [30]Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7132–7141, 2018. doi: 10.1109/CVPR.2018.00745.DOI
- [31]Stefan Kahl, Connor M. Wood, Maximilian Eibl, and Holger Klinck. Birdnet: A deep learning solution for avian diversity monitoring. Ecological Informatics, 61:101236, March 2021. ISSN 1574-9541. doi: 10.1016/j.ecoinf.2021.101236. URL http://dx.doi.org/10.1016/j.ecoinf.2021.101236.
- [32]C. Knapp and G. Carter. The generalized correlation method for estimation of time delay. IEEE Transactions on Acoustics, Speech, and Signal Processing, 24(4):320–327, 1976. doi: 10.1109/TASSP.1976.1162830.DOI
- [33]Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts, 2021. URL https://arxiv.org/abs/2012.07421.
- [34]Yongsen Ma, Gang Zhou, and Shuangquan Wang. Wifi sensing with channel state information: A survey. ACM Comput. Surv., 52(3), June 2019. ISSN 0360-0300. doi: 10.1145/3310194. URL https://doi.org/10.1145/3310194.
- [35]Debapriya Maji, Soye b Nagori, Manu Mathew, and Deepak Poddar. Yolo-pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss, 2022. URL https://arxiv.org/abs/2204.06806.
- [36]Jesse Marshall, Ugne Klibaite, amanda gellis, Diego Aldarondo, Bence Olveczky, and Timothy W Dunn. The pair-r24m dataset for multi-animal 3d pose estimation. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/1ff8a7b5dc7a7d1f0ed65aaa29c04b1e-Paper-round1.pdf.DOI
- [37]Killian Martin, Olivier Adam, Nicolas Obin, and Valérie Dufour. Rookognise: Acoustic detection and identification of individual rooks in field recordings using multi-task neural networks. Ecological Informatics, 72:101818, December 2022. ISSN 1574-9541. doi: 10.1016/j.ecoinf.2022.101818. URL http://dx.doi.org/10.1016/j.ecoinf.2022.101818.
- [38]Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9):1281–1289, August 2018. ISSN 1546-1726. doi: 10.1038/s41593-018-0209-y. URL http://dx.doi.org/10.1038/s41593-018-0209-y.
- [39]Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. Voxceleb: Large-scale speaker verification in the wild. Computer Speech & Language, 60:101027, March 2020. ISSN 0885-2308. doi: 10.1016/j.csl.2019.101027. URL http://dx.doi.org/10.1016/j.csl.2019.101027.
- [40]Hemal Naik, Alex Hoi Hang Chan, Junran Yang, Mathilde Delacoux, Iain D. Couzin, Fumihiro Kano, and Máte Nágy. 3d-pop - an automated annotation approach to facilitate markerless 2d-3d tracking of freely moving birds with marker-based motion capture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21274–21284, June 2023.arxiv.org/abs/2303.13174
- [41]Le Thien Phuc Nguyen, Zhuoran Yu, Khoa Quang Nhat Cao, Yuwei Guo, Tu Ho Manh Pham, Tuan Tai Nguyen, Toan Ngo Duc Vo, Lucas Poon, Soochahn Lee, and Yong Jae Lee. Unitalk: Towards universal active speaker detection in real world scenarios, 2025. URL https://arxiv.org/abs/2505.21954.
- [42]Thi Ngoc Tho Nguyen, Karn N. Watcharasupat, Ngoc Khanh Nguyen, Douglas L. Jones, and Woon-Seng Gan. SALSA-Lite: A fast and effective feature for polyphonic sound event localization and detection with microphone arrays. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9754–9758, 2022. URL https://arxiv.org/abs/2111.08192.
- [43]Talmo D. Pereira, Nathaniel Tabris, Arie Matsliah, David M. Turner, Junyu Li, Shruthi Ravindranath, Eleni S. Papadopayannis, Edna Normand, David S. Deutsch, Z. Yan Wang, Grace C. McKenzie-Smith, Catalin C. Miletut, Marielisa Diez Castro, John D’Uva, Mikhail Kislin, Dan H. Sanes, Sarah D. Kocher, Samuel S.-H. Wang, Annegret L. Falkner, Joshua W. Shaevitz, and Mala Murthy. Sleap: A deep learning system for multi-animal pose tracking. Nature Methods, 19(4):486–495, April 2022. ISSN 1548-7105. doi: 10.1038/s41592-022-01426-1. URL http://dx.doi.org/10.1038/s41592-022-01426-1.
- [45]Nikhil Phaniraj, Kaja Wierucka, Yvonne Zürcher, and Judith M. Burkart. Who is calling? optimizing source identification from marmoset vocalizations with hierarchical machine learning classifiers. Journal of The Royal Society Interface, 20(207), October 2023. ISSN 1742-5662. doi: 10.1098/rsif.2023.0399. URL http://dx.doi.org/10.1098/rsif.2023.0399.
- [46]Archontis Politis, Kazuki Shimada, Parthasaarathy Sudarsanam, Sharath Adavanne, Daniel Krause, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, Yuki Mitsufuji, and Tuomas Virtanen. Starss22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events, 2022. URL https://arxiv.org/abs/2206.01948.
- [47]Yosef Prat, Mor Taub, Ester Pratt, and Yossi Yovel. An annotated dataset of egyptian fruit bat vocalizations across varying contexts and during vocal ontogeny. Scientific Data, 4(1), October 2017. ISSN 2052-4463. doi: 10.1038/sdata.2017.143. URL http://dx.doi.org/10.1038/sdata.2017.143.
- [48]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brunskill, Kyanghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–28518. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/radford23a.html.arxiv.org/abs/2212.04356
- [49]Lukas Rauch, Raphael Schwinger, Moritz Wirth, René Heinrich, Denis Huseljic, Marek Herde, Jonas Lange, Stefan Kahl, Bernhard Sick, Sven Tomforde, and Christoph Scholz. Birdset: A large-scale dataset for audio classification in avian bioacoustics, 2024. URL https://arxiv.org/abs/2403.10380.
- [50]Francisco Romero-Ferrero, Mattia G. Bergomi, Robert C. Hinz, Francisco J. H. Heras, and Gonzalo G. de Polavieja. idtracker.ai: tracking all individuals in small or large collectives of unmarked animals. Nature Methods, 16(2):179–182, January 2019. ISSN 1548-7105. doi: 10.1038/s41592-018-0295-5. URL http://dx.doi.org/10.1038/s41592-018-0295-5.
- [51]Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, and Caroline Pantofaru. Ava-activespeaker: An audio-visual dataset for active speaker detection, 2019. URL https://arxiv.org/abs/1901.01342.
- [52]Linus Rüttimann, Yuhang Wang, Jörg Rychen, Tomas Tomka, Heiko Hörster, and Richard H. R. Hahnloser. Multimodal system for recording individual-level behaviors in songbird groups. PeerJ, 13:e20203, November 2025. ISSN 2167-8359. doi: 10.7717/peerj.20203. URL http://dx.doi.org/10.7717/peerj.20203.
- [53]Neville Ryant, Kenneth Church, Christopher Cieri, Alejandrina Cristia, Jun Du, Sriram Ganapathy, and Mark Liberman. The second dihard diarization challenge: Dataset, task, and baselines, 2019. URL https://arxiv.org/abs/1906.07839.
- [54]Julian C. Schäfer-Zimmermann, Vlad Demartsev, Baptiste Averly, Kiran L. Dhanjal-Adams, Mathieu Dutiel, Gabriella Gall, Marius Faiß, Lily Johnson-Ulrich, Dan Stowell, Marta B. Manser, Marie A. Roch, and Ariana Strandburg-Peshkin. animal2vec and <scp>meerkat</scp>: a self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics. Methods in Ecology and Evolution, 17(3):875–888, December 2025. ISSN 2041-210X. doi: 10.1111/2041-210x.70218. URL http://dx.doi.org/10.1111/2041-210x.70218.
- [55]Kazuki Shimada, Yuichiro Koyama, Shusuke Takahashi, Naoya Takahashi, Emiru Tsunoo, and Yuki Mitsufuji. Multi-acdoa: Localizing and detecting overlapping sounds from the same class with auxiliary duplication permutation invariant training. ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 316–320, 2022.
- [56]Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A. Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, Tuomas Virtanen, and Yuki Mitsufuji. Starrs23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events. In A. Oh, T. Naumann, A. Globerston, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 72931–72957. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/e6c9671ed3b3106b71cafda3ba225c1a-Paper-Datasets_and_Benchmarks.pdf.
- [57]Yusun Shul and Jung-Woo Choi. Cst-former: Transformer with channel-spectro-temporal attention for sound event localization and detection. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8686–8690, 2024. doi: 10.1109/ICASSP48485.2024.10447181.DOI
- [58]Max L Sterling, Ruben Teunisse, and Bernhard Englitz. Rodent ultrasonic vocal interaction resolved with millimeter precision using hybrid beamforming. eLife, 12, 2023. ISSN 2050-084X. doi: 10.7554/eLife.86126. URL http://dx.doi.org/10.7554/eLife.86126.

