betaFeedback

WorksComputer Science

MarineSet: Large-Scale Underwater Acoustic Data Curation and evaluation

Abstract

The scarcity of labeled underwater acoustic data limits the development of data-driven models for marine acoustic monitoring, motivating the use of self-supervised learning to exploit the abundance of unlabeled recordings. However, large-scale data alone does not guarantee effective representation learning, because model performance is strongly influenced by the distribution of the training data. To facilitate such training data, this study introduces a scalable curation pipeline that curates 3,000 hours from 41 years of raw NOAA hydrophone recordings, producing MarineSet, a large-scale and diverse dataset designed for underwater acoustic learning. A baseline model, MarineNet, trained on MarineSet achieves competitive performance on marine mammal call and ship-type classification, highlighting the value of systematic data curation for developing self-supervised learning models in underwater acoustics.

Introduction

The ocean has a diverse soundscape, however, the increasing level of anthropogenic noise threatens the sustainability of our ocean environment. This makes quantifying sound pollution and monitoring marine biodiversity crucial to protecting our ocean environment [1]. For this purpose, various Passive Acoustic Monitoring (PAM) systems are placed worldwide, recording various ocean soundscapes for a long period of time [2, 3, 4]. The resulting large quantity of recordings creates the potential for automated analysis using Machine Learning (ML) [1]. While supervised learning has achieved the best performance in terrestrial bioacoustics [5], the scarcity of labeled underwater recordings makes this approach infeasible at the scale required for general-purpose models. Self-Supervised Learning (SSL) has already shown promise in underwater acoustics [6, 7, 8, 9], providing a viable approach for developing large foundation models from unlabeled recordings. However, these models were not trained on large, curated underwater acoustic datasets, despite the necessity of sufficient scale, diversity, and balance for SSL generalization [10]. High-quality data curation is therefore essential [11, 12, 13]. Motivated by this need, this study introduces MarineSet, a large-scale curated dataset constructed from a substantial volume of web-scraped underwater acoustic recordings collected by hydrophones deployed across U.S. waters. The extensive data coverage of MarineSet enables the representation of a broad range of underwater soundscapes, providing a foundation for developing SSL models for automatic underwater acoustic analysis. To construct MarineSet, a previously introduced curation strategy [13] is applied to a substantially larger volume of recordings. This strategy combines Automatic Identification System (AIS) based vessel curation with unsupervised clustering-based sampling to obtain a balanced representation of the available recordings. An overview of the resulting pipeline is shown in Figure 1.1 To establish a baseline, MarineNet is trained on the complete MarineSet dataset. MarineNet achieves competitive performance compared with existing underwater acoustic models, demonstrating the potential of MarineSet for SSL.

Figure 1. Overview of the proposed data curation pipeline.

Experimental setup

Web-scraped data

National Oceanic and Atmospheric Administration (NOAA) provides a large-scale repository of freely accessible underwater acoustic recordings from hydrophones distributed across U.S. waters [3]. The geographic diversity of these recordings, spanning environments from Hawaii to the Arctic, provides a suitable basis for constructing a diverse curated dataset. From the complete archive, all hydrophones with recordings from 2019 onward were selected, resulting in 48 hydrophones (Figure 2) and a total duration of 41 years, 2 months, 11 days, 12 hours, 19 minutes, and 38 seconds.

Figure 2. Hydrophone locations of the web-scraped data.

AIS based data curation

NOAA provides publicly available AIS data (NOAA AIS source) covering U.S. territorial waters, enabling alignment with underwater acoustic recordings. Since ship-radiated noise (SR-N) can exhibit low contrast with background noise, a clustering-based approach alone may fail to preserve vessel recordings. AIS information is therefore incorporated to identify vessel-present recordings and ensure vessel coverage during curation. As AIS data was unavailable for some locations, this process was limited to 28 hydrophones. For each hydrophone, a 4×44 \times4 km recording range centered on the hydrophone was defined (similar to [14]), and all vessels transmitting an AIS message within this area were considered recorded. This alignment provides weak labels for curation, indicating that a vessel was within the defined range, without confirming that its acoustic signal was captured. This distinction is acceptable for curation, where the objective is to retain vessel diversity rather than assign precise acoustic labels. To reduce vessel imbalance, a duration threshold tt is applied per vessel. All recordings are retained for vessels with total duration below tt, while recordings from vessels exceeding tt are sampled with probability inversely proportional to their total duration. The complete procedure is given in Algorithm 1. The resulting AIS-curated dataset is additionally annotated with vessel Direction of Arrival (DOA), estimated every 5 s by interpolating vessel positions between AIS transmissions.

Algorithm 1 AIS curation algorithm
# X: raw PAM recordings
# A: AIS data
# t: threshold value
X_AIScurated = []
X_ships = align_AIS_with_audio(X, A)
vessel_count = extract_occurrence_unique_vessels(X_ships)
vessel_count[vessel_count < t] = t
vessel_prob = t / vessel_count
for vessel, audio in X_ships do
    vessel_id = get_index(vessel, vessel_count)
    if random.random() < vessel_prob[vessel_id] then
        X_AIScurated.append(audio)
        break
    end if
end for

Clustering-based data curation

The clustering-based curation of recordings without AIS information follows the approach proposed in [10]. Since the web-scraped NOAA dataset already provides large-scale and diverse recordings through numerous hydrophones across different ocean environments, the clustering step focuses on balancing the data distribution. Samples are selected from an approximately uniform distribution UU over the support of the original distribution PP. Raw recordings are embedded using the Conformer model from [9] and clustered with hierarchical kk-means (HK-means), where each layer clusters the centroids of the previous layer. To prevent the exponential reduction in samples across layers, rtr_t points closest to each centroid are re-sampled as input to the next layer. Repeated quantization progressively flattens the centroid distribution, approximating a uniform distribution [10] (see Algorithm 2). For NOAA curation, HK-means was configured with four layers and kk values in [6000, 400, 40, 10], with rt=[2200,8,5,2]r_t = [2200, 8, 5, 2] and m=10m = 10 resampling iterations. The model was then trained on the complete web-scraped NOAA dataset using a streaming implementation to handle its scale [15]. Samples were selected through top-down hierarchical sampling, excluding samples that were already sampled during the AIS data curation. To approximate uniform sampling across clusters of different sizes, the number of selected samples per cluster is determined by finding the integer nn that minimizes ∣N−∑jkmin⁡(n,sj)∣|N - \sum_{j}^{k} \min(n, s_j)|, where NN is the target size of MarineSet and sjs_j is the size of cluster jj. This results in all samples being selected from smaller clusters and at most nn samples from larger clusters. The optimal nn is found using binary search, and samples closest to the centroids are selected. The target size NN was set to 3,000 hours minus the duration of the AIS-curated dataset. The AIS-curated dataset is combined with clustering-based curated dataset to define MarineSet.

Algorithm 2 Hierarchical kk-means with resampling originated from [10]

Input: Data X∈Rnd×dX \in\mathbb{R}^{n_d \times d}, number of levels TT, number of clusters per level (kt)1≤t≤T(k_t)_{1\le t\le T}, number of resamplings mm, number of points resampled per cluster (rt)1≤t≤T(r_t)_{1\le t\le T}.

Result: A hierarchy of clusters on data: centroids (Ct)1≤t≤T(C_t)_{1\le t\le T} and clusters ((Lt(i))1≤i≤kt)1≤t≤T((L_t^{(i)})_{1\le i\le k_t})_{1\le t\le T}.

for t=1t = 1 to TT do

if t=1t = 1 then I←XI \leftarrow X else I←Ct−1I \leftarrow C_{t-1}

Ct←kmeans(I,kt)C_t \leftarrow kmeans(I, k_t)

LT←assign(I,Ct)L_T \leftarrow assign(I, C_t)

resampling-clustering

for s=1s = 1 to mm do

R←⋃i=1ktresample(Lt(i),rt)R \leftarrow\bigcup_{i=1}^{k_t} resample(L_t^{(i)}, r_t)

Ct←kmeans(R,kt)C_t \leftarrow kmeans(R, k_t)

Lt←assign(I,Ct)L_t \leftarrow assign(I, C_t)

end for

end for

MarineNet: Baseline

As a baseline, MarineNet is trained on the complete MarineSet (AIS- and clustering-based curation) using a randomly initialized Wav2Vec 2.0 architecture [16]. The model is adapted to underwater acoustics through self-supervised training with the Wav2Vec 2.0 objective. It is evaluated on ShipsEar [17], DeepShip [14], Watkins [18], and DCLDE 2026 [19] using a frozen encoder and a linear sigmoid classifier trained with binary cross-entropy. Since no SSL model has been trained on a curated underwater acoustic dataset, MarineNet is compared with supervised models Google Whale [20] and SurfPerch [21]. Performance is measured using mean Average Precision (mAP) and weighted ROC-AUC.

Results

AIS based data Curation

The distribution of unique MMSIs (Maritime Mobile Service Identity), corresponding to vessel IDs, before curation is shown in Figure 3, revealing a strong imbalance in which a small number of vessels dominate the recordings. The optimal threshold-value tt was selected at the knee of this distribution, corresponding to t=250t = 250. After AIS-based curation, the dataset contains 968.2 hours of recordings from 6,540 unique MMSIs across 28 individual hydrophones. As shown in Figure 3, the curation process substantially reduces the dominance of frequently recorded vessels, resulting in a more balanced vessel distribution while maintaining large-scale coverage and diversity.

Figure 3. AIS aligned data distribution, with the original distribution in blue and the curated distribution in orange. toptt_{\mathrm{opt}} is visualized as a red line.

Clustering-based data curation

The resulting curated dataset contains 2,031.8 hours of recordings from 43 hydrophones, geographically distributed across U.S. waters (Figure 4). In contrast to the original geographical distribution (Figure 2), previously overrepresented areas are no longer dominant, resulting in a more balanced geographic distribution. The hydrophones span diverse environments, from the Arctic to the Gulf of Mexico, preserving broad marine acoustic diversity despite the rebalancing process. Seasonal and diurnal variation is also represented (Figure 5), with recordings covering all four seasons and an approximately uniform distribution across the day, with summer being slightly overrepresented.

Geographical distribution of recordings selected through clustering-based curation

Figure 4. Geographical distribution of recordings selected through clustering-based curation. Colors indicate the total duration of selected recordings in seconds.

Figure 5. Distribution of recordings selected through clustering-based curation, with seasons shown on the left and time of day on the right. The size of each quarter represents the total duration of selected recordings in seconds.

Underwater Acoustic Classification

To evaluate MarineNet as a baseline model, its performance is assessed on ship-type classification and on marine mammal call classification (see Table 1). MarineNet demonstrates competitive performance relative to supervised models, outperforming Google Whale on both DeepShip and Watkins. However, both Google Whale and MarineNet show limited performance on the multilabel classification task represented by DCLDE2026. SurfPerch consistently achieves superior performance, which is expected given that it was trained on a large, labeled underwater acoustic dataset specifically designed for marine life classification.

DatasetGoogle mAPGoogle AUCWhale mAPWhale AUCSurfPerch mAPSurfPerch AUCMarineNet mAPMarineNet AUC
ShipsEar0.560.750.620.820.540.74
Deepship0.460.740.600.820.550.78
Watkin’s0.550.960.890.990.650.96
DCLDE 2026 Species0.530.860.760.950.420.81

Table 1. Classification performance underwater on acoustic tasks.

Discussion

The proposed data curation pipeline demonstrates how a large, diverse, and balanced dataset can be extracted from massive volumes of web-scraped underwater acoustic data. Through unsupervised clustering, the method captures geographic and seasonal variability without requiring manual annotations. The resulting MarineSet captures a broad range of underwater acoustic soundscapes across U.S. waters. This diversity extends beyond existing datasets, which often focus on specific marine mammal species or contain limited amounts of ship-radiated noise data, making MarineSet suitable for a broad range of underwater acoustic applications. MarineSet also provides a foundation for self-supervised learning (SSL), as demonstrated by the competitive performance of the baseline MarineNet model on both ship-type classification and marine mammal call classification tasks. While our baseline MarineNet model demonstrates the viability of SSL on MarineSet, the dataset itself, not the particular model, is the primary contribution. MarineNet serves as a proof-of-concept that curated, diverse, unsupervised data can support competitive underwater acoustic modeling. Although supervised approaches are generally known to outperform SSL-based models in bioacoustics [5], MarineNet achieves performance approaching that of supervised underwater acoustic models, despite serving just as a baseline model. This highlights the potential of MarineSet to support the development of SSL approaches for a diverse range of underwater acoustic tasks, including marine mammal and ship monitoring, underwater sound pollution assessment, and biodiversity monitoring. Future research will focus on the development of an optimized SSL model on MarineSet in continuation on the baseline MarineNet presented.

Conclusion

Underwater acoustics suffers from a scarcity of labeled datasets, limiting supervised learning for marine monitoring. Self-supervised learning (SSL) offers an alternative, but its application to underwater sound classification has been limited by the lack of suitable large-scale datasets. This work addresses this gap with an unsupervised curation pipeline that produces MarineSet, a large, diverse, and balanced underwater acoustic dataset designed for SSL. MarineNet, trained on MarineSet, achieves competitive performance with supervised approaches on ship-type and marine mammal call classification. These results demonstrate that curated unlabeled data can enable effective SSL for underwater acoustic monitoring without costly manual annotation. MarineSet provides a foundation for developing robust SSL models for applications including marine sound pollution assessment and biodiversity monitoring.

References

  1. [1]H.I. Hummel, R.D. van der Mei, and S. Bhulai, “A survey on machine learning in ship radiated noise,” Ocean Engineering, vol. 298, pp. 117252, April 2024.DOI
  2. [2]M.J.G. Parsons, T. Lin, T.A. Mooney, C. Erbe, F. Juanes, M. Lammers, S. Li, et al., “Sounding the call for a global library of underwater biological sounds,” Frontiers in Ecology and Evolution, vol. 10, pp. 810156, February 2022.
  3. [3]National Oceanic and Atmospheric Administration, “Passive-acoustic-data,” https://www.noaa.gov/products/passive-acoustic-data, 2017, Accessed: 9 December 2025.
  4. [4]Ocean Network Canada, “Dataportal,” https://data.oceannetworks.ca/home, 2007, Accessed: 9 December 2025.
  5. [5]Bart van Merriënboer, Vincent Dumoulin, Jenny Hamer, Lauren Harrell, Andrea Burns, and Tom Denton, “Perch 2.0: The better lesson for bioacoustics,” arXiv preprint arXiv:2508.04665, 2025.arxiv.org/abs/2508.04665
  6. [6]W. Li, Z. Chen, J. Wang, T. Liu, H. Chen, and G. Xu, “A cross-architecture masked contrastive learning framework for few-shot underwater acoustic target classification,” Knowledge-Based Systems, p. 114252, October 2025.DOI
  7. [7]K. Xu, Q. Xu, K. You, B. Zhu, M. Feng, D. Feng, and B. Liu, “Self-supervised learning–based underwater acoustical signal classification via mask modeling,” The Journal of the Acoustical Society of America, vol. 154, no. 1, pp. 5–15, July 2023.DOI
  8. [8]Q. Xu, J. Jiang, K. Xu, Y. Dou, C. Gao, B. Zhu, K. You, and Q. Zhu, “Self-supervised learning-for underwater acoustic signal classification with mixup,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 17, pp. 3530–3542, October 2023.DOI
  9. [9]H.I. Hummel, A. Gansekoele, S. Bhulai, and R.D. van der Mei, “The computation of generalized embeddings for underwater acoustic target recognition using contrastive learning,” Applied Acoustics, vol. 243, pp. 111103, February 2026.
  10. [10]H.V. Vo, V. Khalidov, T. Darcet, T. Moutakanni, N. Smetanin, M. Szafraniec, H. Touvron, et al., “Automatic data curation for self-supervised learning: A clustering-based approach,” arXiv preprint arXiv:2405.15613, 2024.arxiv.org/abs/2405.15613
  11. [11]H. Xu, S. Xie, X.E. Tan, P. Huang, R. Howes, V. Sharma, S. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer, “Demystifying clip data,” in Proc. International Conference on Learning Representations, 2024, pp. 47812–47831.arxiv.org/abs/2309.16671
  12. [12]C. Jose, T. Moutakanni, D. Kang, F. Baldassarre, T. Darcet, H. Xu, D. Li, et al., “Dinov2 meets text: A unified framework for image-and pixel-level vision-language alignment,” in Proc. Computer Vision and Pattern Recognition Conference, 2025, pp. 24905–24916.arxiv.org/abs/2412.16334
  13. [13]H.I. Hummel, S. Bhulai, B. Ghani, and R.D. van der Mei, “Automated data curation for self-supervised learning in underwater acoustic analysis,” in Proc. 11th Convention of the European Acoustics Association: Forum Acusticum / EuroNoise 2025, May 2025, pp. 2165–2170.arxiv.org/abs/2505.20066
  14. [14]M. Irfan, Z. Jiangbin, S. Ali, M. Iqbal, Z. Masood, and U. Hamid, “Deepship: An underwater acoustic benchmark dataset and a separable convolution based autoencoder for classification,” Expert Systems with Applications, vol. 183, pp. 115270, November 2021.DOI
  15. [15]D. Sculley, “Web-scale k-means clustering,” in Proc. of the 19th international conference on World wide web, 2010, pp. 1177–1178.DOI
  16. [16]A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. 34th Conference on Neural Information Processing Systems, 2020, vol. 33, pp. 12449–12460.arxiv.org/abs/2006.11477
  17. [17]D. Santos-Domínguez, S. Torres-Guijarro, A. Cardenal-López, and A. Pena-Gimenez, “Shipsear: An underwater vessel noise database,” Applied Acoustics, vol. 113, pp. 64–69, December 2016.
  18. [18]L. Savigh, M. Daher, J. Allen, H. Gordon, K. Joyce, C. Stuhlmann, and P. Tyack, “The watkins marine mammal sound database: an online, freely accessible resource,” in Proc. of Meetings on Acoustics, February 2017, vol. 27, p. 040013.
  19. [19]K.J. Palmer, E. Cummings, M.G. Dowd, K. Frasier, F. Frazao, A. Harris, A. Houweling, et al., “A public dataset of annotated orcinus orca acoustic signals for detection and ecotype classification,” Scientific data, vol. 12, no. 1, pp. 1137, July 2025.DOI
  20. [20]L. Harrell, “Whistles, songs, boings, and biotwangs: Recognizing whale vocalizations with ai,” https://research.google/blog/whistles-songs-boings-and-biotwangs-recognizing-whale-vocalizations-with-ai/, 2024, Accessed: 7-Oct-2025.
  21. [21]B. Williams, B. van Merriënboer, V. Dumoulin, J. Hamer, E. Triantafillou, A.B. Fleishman, M. McKown, J.E. Munger, et al., “Leveraging tropical reef, bird and unrelated sounds for superior transfer learning in marine bioacoustics,” arXiv preprint arXiv:2404.16436, 2024.arxiv.org/abs/2404.16436

Paper details

vlnc:W_xfhfxf3k
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents