betaFeedback

Reading v1View latest version

WorksMathematics

Bayesian inference on beta diversity via feature allocation models with imperfect detection

Abstract

Beta diversity quantifies variation in species composition across ecological communities and is fundamental for understanding biodiversity patterns across space and environmental gradients. Statistical inference on beta diversity is challenging: species occurrence data are high dimensional, many species remain unobserved despite extensive sampling, and surveys are subject to imperfect detection. Existing approaches are typically based on empirical dissimilarity indices with limited uncertainty quantification or on models that rely on unrealistic exchangeability and perfect detection assumptions. We introduce a new class of Bayesian feature allocation models for partially exchangeable species occurrence data with imperfect detection. The framework combines latent feature allocation models with occupancy-based detection mechanisms, allowing heterogeneous species compositions across sites, explicitly accounting for false negatives, and accommodating the discovery of previously unobserved species. We develop coherent probabilistic inference for species sharing and beta diversity, deriving explicit posterior and predictive distributions for between-community heterogeneity, including the number of shared species across sites and the number expected under future sampling. These analytical results yield interpretable posterior summaries of compositional heterogeneity and facilitate scalable inference in high-dimensional biodiversity studies. Simulation studies and an application to global fungal biodiversity data demonstrate improved inference on species sharing and between-community diversity.

Introduction

Biodiversity studies increasingly seek to characterize how species compositions vary across ecological communities and geographic regions. Quantifying this between-community heterogeneity, commonly referred to as β\beta-diversity, is fundamental for understanding ecological processes, monitoring environmental change, and prioritizing conservation efforts [40]. A central scientific objective is to infer the extent of species sharing across locations and to predict how many previously unobserved species would be discovered through additional sampling. Such questions arise naturally in large-scale biodiversity monitoring projects, where species occurrence data are collected across multiple habitats, ecosystems, or geographic regions.

Statistical inference on β\beta-diversity is challenging for several reasons. Species occurrence data are typically high dimensional and sparse, with many species remaining unobserved even after extensive sampling. Ecological communities often exhibit substantial heterogeneity across locations due to differences in environmental conditions and habitat characteristics, making exchangeability assumptions unrealistic. In addition, biodiversity surveys are affected by imperfect detection: species that are present may fail to be observed because of limitations in sampling technologies and measurement protocols. Ignoring detection error can substantially bias estimates of species richness, species sharing, and between-community heterogeneity. These challenges are particularly pronounced in modern DNA meta-barcoding studies. We are motivated by the Global Spore Sampling Project (GSSP) [27, 1], which collected airborne fungal spore samples at locations around the world using cyclone samplers. Ecologists typically analyze these data in terms of operational taxonomic units (OTUs), obtained by assigning taxonomic labels to DNA sequences [35]. Because the proportion of sequences mapped to each OTU is not a reliable indicator of abundance, meta-barcoding studies commonly reduce the observations to binary occurrence data indicating whether a species is detected in each sample. These data are characterized by large numbers of rare species, substantial heterogeneity across locations, and non-negligible false negative rates arising from low DNA concentrations, PCR biases, and sequencing errors [11, 33].

Despite the importance of inference on β\beta-diversity, existing approaches remain largely based on empirical dissimilarity indices with limited probabilistic interpretation and inadequate uncertainty quantification [31]. Occupancy models explicitly account for imperfect detection by separating ecological and observational processes [22, 32, 41, 39], and have become standard tools in ecology. However, these models are not designed to characterize latent species sharing structure across heterogeneous communities or to accommodate the continual discovery of previously unobserved species in highly diverse ecosystems.

Bayesian nonparametric feature allocation models provide a natural probabilistic framework for species occurrence data. Indian buffet processes [38, 16, 5] and related models [20, 4, 23, 6, 13] characterize binary latent feature matrices in which rows correspond to samples and columns to a potentially unbounded collection of features. In biodiversity applications the features are species, so that new species emerge as sampling proceeds and the latent feature matrix encodes precisely the pattern of species sharing that β\beta-diversity seeks to quantify. Existing models, however, are ill-suited to this task on two counts. First, they assume exchangeability, that is, invariance of the occurrence matrix under permutations of the samples, which precludes the systematic variation in species composition across communities that β\beta-diversity is about. Second, they assume perfect detection, and thus attribute every absence to genuine absence rather than to a failure to observe. Partially exchangeable constructions have recently received growing attention for species sampling models (e.g., [12]; [9]), but substantially fewer results are available for feature allocations: [34] proposed a bivariate beta process for genetic variants across two heterogeneous populations, though the approach is restricted to two groups and does not admit a tractable analytical characterization, while [14] introduced a broad class of priors for partially exchangeable trait allocations, without addressing imperfect detection or developing inferential results for biodiversity functionals such as species sharing and β\beta-diversity.

Motivated by these challenges, we propose the class of Multi-site Occupancy-aware Species Allocation with Imperfect deteCtion (MOSAIC) models, which addresses both issues at once. Partial exchangeability allows species compositions to vary systematically across sites, while an occupancy-based detection layer separates the probability that a species occupies a site from the probability that it is recorded there, so that an absence in the data need not be an absence in the field. The number of species is finite but random, a choice that preserves substantial modeling flexibility while enabling analytical tractability and scalable posterior computation; by contrast, natural extensions of infinite-activity beta process constructions to heterogeneous populations may violate fundamental finiteness conditions or require computationally prohibitive inference procedures ([34]).

The resulting framework supports a coherent probabilistic treatment of diversity at every level. While α\alpha-diversity and γ\gamma-diversity, measuring within-community and global diversity, respectively, have received extensive attention in the Bayesian nonparametric literature ([21]; [10]; [13]), this is not the case for multi-site incidence data, for which even the distribution of the number of distinct species at each site has not been characterized; and substantially less work has addressed principled inference on β\beta-diversity, despite its central ecological importance ([31]). We therefore develop the α\alpha- and γ\gamma-diversity theory first, obtaining prior and posterior distributions for the site-specific and pooled species counts, along with predictive distributions for the species that would be discovered under further sampling. Beyond their own interest, these results underpin our treatment of β\beta-diversity, since the same machinery delivers the distribution of the number of species shared between any pair of sites, the main ingredient of our proposed index. The predictive quantities are moreover directly relevant for biodiversity monitoring and adaptive sampling design, where decisions must be made regarding whether and where further sampling effort should be allocated. The resulting summaries of compositional heterogeneity are interpretable and account simultaneously for latent species richness, partial exchangeability, and imperfect detection. We further specify a hierarchical prior in which site-specific covariates drive the occurrence probabilities, inducing borrowing of information across sites.

The remainder of the paper is organized as follows. Section 2 introduces the proposed class of partially exchangeable feature allocation models with imperfect detection, while Section 3 develops their theoretical properties and studies inference on biodiversity functionals, including β\beta-diversity. Section 4 presents simulation studies evaluating finite-sample performance, while Section 5 analyzes global fungal biodiversity data from the GSSP study.

The MOSAIC model

Notation and model

In this section, we define the notation and introduce our proposed class of feature allocation models for partially exchangeable data with imperfect detection. The data are sampled from QQ distinct sites, indexed by q∈{1,…,Q}q \in\{1,\ldots,Q\}; we denote by nqn_q the number of samples from site qq and by n=(n1,…,nQ)\mathbf{n}=(n_1,\ldots,n_Q) the corresponding vector of sample sizes. We assume a common species reference set across sites, and we denote by (X~j)j≥1(\widetilde{X}_j)_{j\geq1} the sequence of all possible species labels, which are unknown a priori. Let Xq\mathcal{X}_q be the set of distinct species actually observed at site qq and let Kq,nq=∣Xq∣=kqK_{q,n_q}=|\mathcal{X}_q|=k_q be its cardinality. The total number of distinct species observed across all sites is Kn=∣⋃q=1QXq∣=kK_n=\left|\bigcup_{q=1}^{Q}\mathcal{X}_q\right|=k, and we write X1,…,XkX_1,\ldots,X_k for the labels of the species actually observed, which form a subset of the complete list (X~j)j≥1(\widetilde{X}_j)_{j\geq1}. This notation explicitly accounts for the fact that some species may remain unseen. Finally, let Cnq,nr=∣Xq∩Xr∣=cq,rC_{n_q,n_r}=|\mathcal{X}_q\cap\mathcal{X}_r|=c_{q,r} denote the number of species shared between the samples from sites qq and rr with q≠rq\ne r. Because sites may share species, in general Kn≤∑q=1QKq,nqK_n\leq\sum_{q=1}^{Q}K_{q,n_q}; for instance, with Q=2Q=2 we have Kn=K1,n1+K2,n2−Cn1,n2K_n=K_{1,n_1}+K_{2,n_2}-C_{n_1,n_2}. Throughout, we further assume that the species pool is finite, that is, the complete list reduces to X~1,…,X~N\widetilde{X}_1,\ldots,\widetilde{X}_N, so that the number of distinct species observed in the full sample satisfies Kn≤NK_n\leq N. The total number of species NN is itself unknown and is inferred from the data. Accordingly, we adopt the beta–Bernoulli (BB) class of feature allocation models ([16], [13]), under which the distribution of KnK_n has bounded support given NN. This choice is motivated by analytical tractability: known infinite-species models lead to considerably less tractable expressions ([34]).

The data can be encoded as binary occurrence indicators, one for each combination of sample, species, and site, and we distinguish between a version involving the full species list and its observed counterpart. Let Y~=(Y~ijq)\widetilde{\mathbf{Y}}=(\widetilde{Y}_{ijq}) with i=1,…,nqi=1,\ldots,n_q; j=1,…,Nj=1,\ldots,N; q=1,…,Qq=1,\ldots,Q, where Y~ijq=1\widetilde{Y}_{ijq}=1 if species X~j\widetilde{X}_j is recorded in sample ii of site qq, and Y~ijq=0\widetilde{Y}_{ijq}=0 otherwise. A zero entry is ambiguous: species X~j\widetilde{X}_j may be absent from site qq at the time of sampling, or present but not detected. We collect the observed occurrences in Y=(Yijq)\mathbf{Y} = (Y_{ijq}) with i=1,…,nqi = 1,\ldots,n_q; j=1,…,kj = 1,\ldots,k; q=1,…,Qq = 1,\ldots,Q, obtained from Y~\widetilde{\mathbf{Y}} by discarding the species that are never recorded, namely those jj for which Y~ijq=0\widetilde{Y}_{ijq} = 0 for every ii and qq; equivalently, each species retained in Y\mathbf{Y} has at least one non-zero entry. The distinction between Y~\widetilde{\mathbf{Y}} and Y\mathbf{Y} would be inconsequential if NN were known, since the discarded species could then be reinstated, or if all species were discovered, in which case N=kN = k and the two arrays coincide. These are not realistic assumptions in ecological applications, where neither the labels nor the number of undiscovered species is known a priori, and hence Y~\widetilde{\mathbf{Y}} carries strictly more information than Y\mathbf{Y}. In what follows we specify a model for Y~\widetilde{\mathbf{Y}}, which induces a model for the observed indicators Y\mathbf{Y}.

Let ηq=(η1q,…,ηNq)\boldsymbol{\eta}_q = (\eta_{1q},\ldots,\eta_{Nq}) and γq=(γ1q,…,γNq)\boldsymbol{\gamma}_q = (\gamma_{1q},\ldots,\gamma_{Nq}), for q=1,…,Qq = 1,\ldots,Q, collect independent random probabilities representing the species- and site-specific occupancy and detectability components, respectively. In our MOSAIC model, given these parameters, the entries of Y~\widetilde{\mathbf{Y}} are conditionally independent Bernoulli random variables,

Y~ijq∣ηjq,γjq∼indBernoulli⁡(ηjqγjq),(1)\widetilde{Y}_{ijq} \mid\eta_{jq},\gamma_{jq} \mathrel{\overset{\mathrm{ind}}{\sim}} \operatorname{Bernoulli}(\eta_{jq}\gamma_{jq}), \tag*{(1)}

References

  1. [1]Abrego, N., B. Furneaux, B. Hardwick, P. Somervuo, I. Palorinne, C. A. Aguilar-Trigueros, N. R. Andrew, U. V. Babiy, T. Bao, G. Bazzano, et al. (2024). Airborne DNA reveals predictable spatial and seasonal dynamics of fungi. Nature 631 (8022), 835–842.DOI
  2. [2]Anderson, M. J., T. O. Crist, J. M. Chase, M. Vellend, B. D. Inouye, A. L. Freestone, N. J. Sanders, H. V. Cornell, L. S. Comita, K. F. Davies, et al. (2011). Navigating the multiple meanings of β diversity: a roadmap for the practicing ecologist. Ecology Letters 14 (1), 19–28.
  3. [3]Barwell, L. J., N. J. Isaac, and W. E. Kunin (2015). Measuring β-diversity with species abundance data. Journal of Animal Ecology 84 (4), 1112–1122.DOI
  4. [4]Battiston, M., S. Favaro, D. M. Roy, and Y. W. Teh (2018). A characterization of product-form exchangeable feature probability functions. The Annals of Applied Probability 28 (3), 1423–1448.
  5. [5]Broderick, T., J. Pitman, and M. Jordan (2013). Feature allocations, probability functions, and paintboxes. Bayesian Analysis 8 (4), 801–836.arxiv.org/abs/1301.6647
  6. [6]Camerlenghi, F., S. Favaro, L. Masoero, and T. Broderick (2024). Scaled process priors for Bayesian nonparametric estimation of the unseen genetic variation. Journal of the American Statistical Association 119 (545), 320–331.
  7. [7]Cao, Y., T. Xu, X. Yu, Z. Hua, H. Liu, W. Gu, H. Liu, H. Kang, Z. Wei, and Z. Xie (2024). Fungal aerosols over the Northwest Pacific to Arctic ocean: The influence of air mass and environmental factors. Atmospheric Environment 337, 120776.DOI
  8. [8]Carpenter, B., A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell (2017). Stan: A probabilistic programming language. Journal of Statistical Software 76, 1–32.DOI
  9. [9]Colombi, A., R. Argiento, F. Camerlenghi, and L. Paci (2025). How many unseen species are in multiple areas? arXiv preprint arXiv:2502.04122.
  10. [10]De Blasi, P., S. Favaro, A. Lijoi, R. H. Mena, I. Prünster, and M. Ruggiero (2015, 2). Are gibbs-type priors the most natural generalization of the dirichlet process? IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 212–229.DOI
  11. [11]Doi, H., K. Fukaya, S. I. Oka, K. Sato, M. Kondoh, and M. Miya (2019). Evaluation of detection probabilities at the water-filtering and initial pcr steps in environmental DNA metabarcoding using a multispecies site occupancy model. Scientific Reports 9(1), 3581.DOI
  12. [12]Franzolini, B., A. Lijoi, I. Prünster, and G. Rebaudo (2025). Multivariate species sampling models. arXiv preprint arXiv:2503.24004.arxiv.org/abs/2503.24004
  13. [13]Ghilotti, L., F. Camerlenghi, and T. Rigon (2026). Bayesian analysis of product feature allocation models. Journal of the Royal Statistical Society Series B: Statistical Methodology 88(2), 540–566.DOI
  14. [14]Ghilotti, L., F. Camerlenghi, T. Rigon, and M. Guindani (2025). Bayesian nonparametric modeling of multivariate count data with an unknown number of traits. arXiv preprint arXiv:2510.24526.arxiv.org/abs/2510.24526
  15. [15]Gotelli, N. J. and R. K. Colwell (2001). Quantifying biodiversity: procedures and pitfalls in the measurement and comparison of species richness. Ecology Letters 4(4), 379–391.
  16. [16]Griffiths, T. L. and Z. Ghahramani (2011). The Indian buffet process: An introduction and review. Journal of Machine Learning Research 12(4).DOI
  17. [17]Hess, O. C., T. van der Deure, M. Bolander, C. A. Leal Dutra, and J. Z. Shik (2025). The evolution of thermal performance curves in fungi farmed by attine ant mutualists in above-ground or below-ground microclimates. Journal of Evolutionary Biology 38(1), 83–93.DOI
  18. [18]Ho, H.-M., C. Y. Rao, H.-H. Hsu, Y.-H. Chiu, C.-M. Liu, and H. J. Chao (2005). Characteristics and determinants of ambient fungal spores in Hualien, Taiwan. Atmospheric Environment 39(32), 5839–5850.
  19. [19]James, I. R. (1972). Products of independent beta variables with application to Connor and Mosimann’s generalized Dirichlet distribution. Journal of the American Statistical Association 67(340), 910–912.DOI
  20. [20]James, L. F. (2017). Bayesian Poisson calculus for latent feature modeling via generalized Indian buffet process priors. The Annals of Statistics 45(5), 2016–2045.
  21. [21]Lijoi, A., R. H. Mena, and I. Prünster (2007). Bayesian nonparametric estimation of the probability of discovering new species. Biometrika 94, 769–786.DOI
  22. [22]MacKenzie, D. I., J. D. Nichols, G. B. Lachman, S. Droege, J. Andrew Royle, and C. A. Langtimm (2002). Estimating site occupancy rates when detection probabilities are less than one. Ecology 83(8), 2248–2255.
  23. [23]Masoero, L., F. Camerlenghi, S. Favaro, and T. Broderick (2022). More for less: predicting and maximizing genomic variant discovery via Bayesian nonparametrics. Biometrika 109(1), 17–32.
  24. [24]Mauri, L. and D. B. Dunson (2025). Factor pretraining in Bayesian multivariate logistic models. Biometrika 112(4), asaf056.
  25. [25]Ochiai, A. (1957). Zoogeographical studies on the soleoid fishes found in Japan and its neighbouring regions—II. Bulletin of the Japanese Society of Scientific Fisheries 22(9), 526–530. In Japanese.
  26. [26]Orloci, L. (1967). An agglomerative method for classification of plant communities. Journal of Ecology 55, 193–206.DOI
  27. [27]Ovaskainen, O., N. Abrego, B. Furneaux, B. Hardwick, P. Somervuo, I. Palorinne, N. R. Andrew, U. V. Babiy, T. Bao, G. Bazzano, et al. (2024). Global spore sampling project: A global, standardized dataset of airborne fungal DNA. Scientific Data 11(1), 561.
  28. [28]Ovaskainen, O., N. Abrego, P. Halme, and D. B. Dunson (2016). Using latent variable models to identify large networks of species-to-species associations at different spatial scales. Methods in Ecology and Evolution 7(5), 549–555.
  29. [29]Pakpour, S., D.-W. Li, and J. Klironomos (2015). Relationships of fungal spore concentrations in the air and meteorological factors. Fungal Ecology 13, 130–134.
  30. [30]Pollock, K. H., J. D. Nichols, C. Brownie, and J. E. Hines (1990). Statistical inference for capture-recapture experiments. Wildlife Monographs, 3–97.
  31. [31]Rigon, T., C.-L. Hsu, and D. B. Dunson (2025). A Bayesian theory for estimation of biodiversity. arXiv:2502.01333.arxiv.org/abs/2502.01333
  32. [32]Royle, J. A., J. D. Nichols, M. Kéry, and E. Ranta (2005). Modelling occurrence and abundance of species when detection is imperfect. Oikos 110, 353–359.
  33. [33]Schlegel, M., A. D. Treindl, J. Panziera, V. Zengerer, D. Zani, J. Brännhage, and A. Gross (2024). A case study on the application of spore sampling for the monitoring of macrofungi. Molecular Ecology Resources 24(4), e13941.
  34. [34]Shen, Y., L. Masoero, J. G. Schraiber, and T. Broderick (2025). Double trouble: Predicting new variant counts across two heterogeneous populations. Bayesian Analysis 1(1), 1–33.
  35. [35]Somervuo, P., S. Koskela, J. Pennanen, R. Henrik Nilsson, and O. Ovaskainen (2016). Unbiased probabilistic taxonomic classification for DNA barcoding. Bioinformatics 32(19), 2920–2927.DOI
  36. [36]Stolf, F. and D. B. Dunson (2025). Infinite joint species distribution models. Biometrika 112(4), asaf055.
  37. [37]Tedersoo, L., M. Bahram, S. Põlme, U. Kõljalg, N. S. Yorou, R. Wijesundera, L. V. Ruiz, A. M. Vasco-Palacios, P. Q. Thu, A. Suija, et al. (2014). Global diversity and geography of soil fungi. Science 346(6213), 1256688.
  38. [38]Teh, Y. and D. Gorur (2009). Indian buffet processes with power-law behavior. In Advances in Neural Information Processing Systems, Volume 22.
  39. [39]Tobler, M. W., M. Kéry, F. K. Hui, G. Guillera-Arroita, P. Knaus, and T. Sattler (2019). Joint species distribution models with species correlations and imperfect detection. Ecology 100(8), e02754.
  40. [40]Tuomisto, H. (2010). A diversity of beta diversities: straightening up a concept gone awry. Part 1. Defining beta diversity as a function of alpha and gamma diversity. Ecography 33(1), 2–22.
  41. [41]Wenger, S. J. and M. C. Freeman (2008). Estimating species occurrence, abundance, and detection probability using zero-inflated distributions. Ecology 89, 2953–2959.
  42. [42]Whittaker, R. H. (1960). Vegetation of the Siskiyou Mountains, Oregon and California. Ecological Monographs 30(3), 279–338.
  43. [43]Zito, A., T. Rigon, O. Ovaskainen, and D. B. Dunson (2023). Bayesian modeling of sequential discoveries. Journal of the American Statistical Association 118(544), 2521–2532.

Paper details

vlnc:W_qnynet34
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents