betaFeedback

WorksMathematics

Bayesian inference on beta diversity via feature allocation models

Abstract

Beta diversity quantifies variation in species composition across ecological communities and is fundamental for understanding biodiversity patterns across space and environmental gradients. Inference is complicated by incomplete sampling, differences among communities and imperfect detection. We develop a partially exchangeable Bayesian feature allocation model for species occurrence data that accommodates unobserved species and incorporates site-specific covariates to relate variation in species composition to environmental conditions. The model yields explicit conditional distributions for species richness and sharing, together with predictions of the species newly recorded at individual sites or newly shared between sites under additional sampling. We also introduce a novel probabilistic beta diversity index with posterior uncertainty quantification. An occupancy–detection representation preserves the tractability of the occurrence model and allows sensitivity analysis under specified detectability assumptions. This tractability makes the approach scalable, enabling inference in large biodiversity studies. Simulation studies assess predictive performance, and an application to global airborne fungal data illustrates how accounting for unobserved species and borrowing information across sites affect inference on compositional heterogeneity.

Introduction

Biodiversity studies increasingly seek to characterize how species compositions vary across ecological communities and geographic regions. Quantifying this between-community heterogeneity, commonly referred to as β\beta-diversity, is fundamental for understanding ecological processes, monitoring environmental change, and prioritizing conservation efforts [39]. A central scientific objective is to infer the extent of species sharing across locations and to predict how many previously unobserved species would be discovered through additional sampling. Such questions arise naturally in large-scale biodiversity monitoring projects, where species occurrence data are collected across multiple habitats, ecosystems, or geographic regions.

Statistical inference on β\beta-diversity is challenging for several reasons. Species occurrence data are typically high dimensional and sparse, with many species remaining unobserved even after extensive sampling. Ecological communities often exhibit substantial heterogeneity across locations due to differences in environmental conditions and habitat characteristics, making exchangeability assumptions unrealistic. In addition, biodiversity surveys are affected by imperfect detection: species that are present may fail to be observed because of limitations in sampling technologies and measurement protocols. Occupancy models are a standard tool in ecology for addressing imperfect detection by separating ecological and observational processes [22, 31, 38]. However, standard formulations do not jointly address species sharing across heterogeneous communities and the discovery of previously unobserved species. These challenges are particularly pronounced in DNA meta-barcoding studies.

We are motivated by the Global Spore Sampling Project (GSSP) [26, 1], which collected airborne fungal spore samples at locations around the world using cyclone samplers. Ecologists typically analyze these data in terms of operational taxonomic units (OTUs), obtained by assigning taxonomic labels to DNA sequences [34]. Because the proportion of sequences mapped to each OTU is not a reliable indicator of abundance, metabarcoding studies commonly reduce the observations to binary occurrence data indicating whether a species is detected in each sample. These data are characterized by large numbers of rare species, substantial heterogeneity across locations, and non-negligible false negative rates arising from low DNA concentrations, PCR biases, and sequencing errors [11, 32].

Despite the importance of inference on β\beta-diversity, existing approaches remain largely based on empirical dissimilarity indices with limited probabilistic interpretation and inadequate uncertainty quantification [30]. Differences in observed species overlap may reflect incomplete sampling rather than genuine variation in community composition, since species not yet recorded may nevertheless be shared across communities. The inferential goal is therefore to characterize the underlying species-sharing structure while accounting for incomplete sampling. This calls for a probabilistic framework that jointly accommodates heterogeneity across communities, incomplete sampling and imperfect detection.

Bayesian nonparametric feature allocation models provide a natural probabilistic framework for species occurrence data. Indian buffet processes [37, 16, 5] and related models [20, 4, 6, 13] (Masaero et al. 2022) characterize binary latent feature matrices in which rows correspond to samples and columns to a potentially unbounded collection of features. In biodiversity applications the features are species, so that new species emerge as sampling proceeds and the latent feature matrix encodes precisely the pattern of species sharing that β\beta-diversity seeks to quantify. Exchangeable formulations, however, do not capture systematic variation in species composition across communities. Moreover, formulations without a detection layer attribute every zero to absence, and those without community-specific covariates cannot directly relate compositional heterogeneity to environmental conditions. Partially exchangeable constructions have recently received growing attention for species sampling models (e.g., [12, 9]), but substantially fewer results are available for feature allocations: Shen et al. [33] proposed a bivariate beta process for genetic variants across two heterogeneous populations, though the approach is restricted to two groups and does not admit a tractable analytical characterization, while Ghilotti et al. [14] introduced a broad class of priors for partially exchangeable trait allocations, without addressing imperfect detection, including group-specific covariates or developing inferential results for biodiversity functionals such as species sharing and β\beta-diversity.

Motivated by these challenges, we propose the class of Multi-site Occupancy-aware Species Allocation with Imperfect detecTion (MOSAIC) models, which addresses all the above issues at once. Partial exchangeability allows species compositions to vary systematically across sites, while an occupancy-based detection layer distinguishes presence at the time of sampling from detection, so that a zero in the data need not imply absence in the field. A hierarchical prior allows site-specific covariates to drive species occurrence probabilities, linking compositional heterogeneity to environmental conditions while enabling borrowing of information across sites. The number of species is finite but random, a choice that preserves substantial modeling flexibility while enabling analytical tractability and scalable posterior computation; by contrast, natural extensions of infinite-activity beta process constructions to heterogeneous populations may violate fundamental finiteness conditions or require computationally prohibitive inference procedures ([33]). Our motivating application illustrates inference with more than 15,000 observed species across over 30 sites.

The resulting framework supports a coherent probabilistic treatment of diversity at every level: within-community (α\alpha), global (γ\gamma) and between-community (β\beta) diversity. While α\alpha- and γ\gamma-diversity have received extensive attention in the Bayesian nonparametric literature ([21, 10, 13]), species sharing in heterogeneous incidence data calls for a joint treatment across sites. We first develop the α\alpha- and γ\gamma-diversity theory, deriving prior and posterior distributions for the site-specific and pooled species counts, together with predictions of discoveries under further sampling. These results underpin the characterization of the prior and posterior number of species shared between sites. Specifically, we derive a four-component predictive decomposition of the increase in overlap under additional sampling. We also define a β\beta-diversity index that normalizes the expected between-site co-occurrence by the geometric mean of the within-site co-occurrences. The predictive quantities are directly relevant for biodiversity monitoring and adaptive sampling design, where decisions must be made regarding whether and where further sampling effort should be allocated.

The MOSAIC model

Notation and model

We consider data sampled from QQ distinct sites, indexed by q∈{1,…,Q}q \in\{1,\ldots,Q\}; we denote by nqn_q the number of samples from site qq and by n=(n1,…,nQ)\boldsymbol{n}=(n_1,\ldots,n_Q) the corresponding vector of sample sizes. We assume a common species reference set across sites, and we denote by (X~j)j≥1(\widetilde{X}_j)_{j\ge1} the sequence of all possible species labels, which are unknown a priori. Let Xq\mathcal{X}_q be the set of distinct species actually observed at site qq and let Kq,nq=∣Xq∣=kqK_{q,n_q}=|\mathcal{X}_q|=k_q be its cardinality. The total number of distinct species observed across all sites is Kn=∣⋃q=1QXq∣=kK_n=\left|\bigcup_{q=1}^{Q}\mathcal{X}_q\right|=k, and we write X1,…,XkX_1,\ldots,X_k for the labels of the species actually observed, which form a subset of the complete list (X~j)j≥1(\widetilde{X}_j)_{j\ge1}. Let Cnq,nr=∣Xq∩Xr∣=cq,rC_{n_q,n_r}=|\mathcal{X}_q\cap\mathcal{X}_r|=c_{q,r} denote the number of species shared between the samples from sites qq and rr with q≠rq\ne r. Because sites may share species, in general Kn≤∑q=1QKq,nqK_n\le\sum_{q=1}^{Q}K_{q,n_q}; for instance, with Q=2Q=2 we have Kn=K1,n1+K2,n2−Cn1,n2K_n=K_{1,n_1}+K_{2,n_2}-C_{n_1,n_2}.

Throughout, we further assume that the species pool is finite, that is, the complete list reduces to X~1,…,X~N\widetilde{X}_1,\ldots,\widetilde{X}_N, so that the number of distinct species observed in the full sample satisfies Kn≤NK_n\le N. The total number of species NN is itself unknown and is inferred from the data. Accordingly, we adopt the beta–Bernoulli (BB) class of feature allocation models [16, 13], under which the distribution of KnK_n has bounded support given NN. This choice is motivated by analytical tractability: known infinite-species models lead to considerably less tractable expressions [33].

The data can be encoded as binary occurrence indicators, one for each combination of sample, species, and site, and we distinguish between a version involving the full species list and its observed counterpart. Let Y~=(Y~ijq)\widetilde{Y}=(\widetilde{Y}_{ijq}) with i=1,…,nqi=1,\ldots,n_q; j=1,…,Nj=1,\ldots,N; q=1,…,Qq=1,\ldots,Q, where Y~ijq=1\widetilde{Y}_{ijq}=1 if species X~j\widetilde{X}_j is recorded in sample ii of site qq, and Y~ijq=0\widetilde{Y}_{ijq}=0 otherwise. A zero entry is ambiguous: species X~j\widetilde{X}_j may be absent from site qq at the time of sampling, or present but not detected. We collect the observed occurrences in Y=(Yijq)Y=(Y_{ijq}) with i=1,…,nqi=1,\ldots,n_q; j=1,…,kj=1,\ldots,k; q=1,…,Qq=1,\ldots,Q, obtained from Y~\widetilde{Y} by discarding the species that are never recorded, namely those jj for which Y~ijq=0\widetilde{Y}_{ijq}=0 for every ii and qq; equivalently, each species retained in YY has at least one non-zero entry. The number of discarded species is unknown; if all species had been discovered, N=kN=k and the two arrays would coincide. We specify a model for Y~\widetilde{Y} and derive the induced law of the observed array YY.

Let ηq=(η1q,…,ηNq)\boldsymbol{\eta}_q=(\eta_{1q},\ldots,\eta_{Nq}) and γq=(γ1q,…,γNq)\boldsymbol{\gamma}_q=(\gamma_{1q},\ldots,\gamma_{Nq}), for q=1,…,Qq=1,\ldots,Q, collect independent random probabilities representing the species- and site-specific occupancy and detectability components, respectively. In our MOSAIC model, given these parameters, the entries of Y~\widetilde{Y} are conditionally independent”】【 Bernoulli random variables,

Y~ijq∣ηjq,γjq∼indBernoulli⁡(ηjqγjq),(1)\widetilde{Y}_{ijq} \mid\eta_{jq},\gamma_{jq} \mathrel{\overset{\mathrm{ind}}{\sim}} \operatorname{Bernoulli}(\eta_{jq}\gamma_{jq}), \tag*{(1)}

for i=1,…,nqi=1,\ldots,n_q, j=1,…,Nj=1,\ldots,N and q=1,…,Qq=1,\ldots,Q. The factorization in (1) separates the occupancy and detection components of the model: ηjq\eta_{jq} is the probability that species X~j\widetilde{X}_j is present at site qq when a sample is collected, and γjq\gamma_{jq} is the probability that it is detected, given presence. The two differ because a species present at a site may go unrecorded, so that the observed occurrences are a thinned version of the underlying presence/absence pattern. This is made precise by a data augmentation, in which each Y~ijq\widetilde{Y}_{ijq} is written as the product of two independent Bernoulli random variables representing the occupancy and detection events separately. Let W~=(W~ijq)\widetilde{W}=(\widetilde{W}_{ijq}) and T~=(T~ijq)\widetilde{T}=(\widetilde{T}_{ijq}), with i=1,…,nqi=1,\ldots,n_q, j=1,…,Nj=1,\ldots,N and q=1,…,Qq=1,\ldots,Q, be augmented binary arrays encoding respectively the true presence of each species and its potential detection, so that W~ijq∣ηjq∼indBernoulli⁡(ηjq)\widetilde{W}_{ijq}\mid\eta_{jq}\mathrel{\overset{\mathrm{ind}}{\sim}}\operatorname{Bernoulli}(\eta_{jq}) and T~ijq∣γjq∼indBernoulli⁡(γjq)\widetilde{T}_{ijq}\mid\gamma_{jq}\mathrel{\overset{\mathrm{ind}}{\sim}}\operatorname{Bernoulli}(\gamma_{jq}). Then Y~ijq=W~ijqT~ijq\widetilde{Y}_{ijq}=\widetilde{W}_{ijq}\widetilde{T}_{ijq}, so that a species is recorded if and only if it is both present and detected.

To accommodate unseen species, we treat the total species pool size as random, with N∼Poisson⁡(λ)N\sim\operatorname{Poisson}(\lambda) and λ>0\lambda>0. Moreover, we assume that the parameters γjq\gamma_{jq} and ηjq\eta_{jq} are conditionally independent for j=1,…,Nj=1,\ldots,N and q=1,…,Qq=1,\ldots,Q given a set of parameters (αq,θq,σq)(\alpha_q,\theta_q,\sigma_q) and follow a beta distribution. We will further denote the corresponding vectors of parameters as α=(α1,…,αQ)\boldsymbol{\alpha}=(\alpha_1,\ldots,\alpha_Q), θ=(θ1,…,θQ)\boldsymbol{\theta}=(\theta_1,\ldots,\theta_Q) and σ=(σ1,…,σQ)\boldsymbol{\sigma}=(\sigma_1,\ldots,\sigma_Q). This implies the following specification for the MOSAIC model with parameters (λ,α,θ,σ)(\lambda,\boldsymbol{\alpha},\boldsymbol{\theta},\boldsymbol{\sigma})

References

  1. [1]Abrego, N., B. Furneaux, B. Hardwick, P. Somervuo, I. Palorinne, C. A. Aguilar-Trigueros, N. R. Andrew, U. V. Babiy, T. Bao, G. Bazzano, et al. (2024). Airborne DNA reveals predictable spatial and seasonal dynamics of fungi. Nature 631 (8022), 835–842.DOI
  2. [2]Anderson, M. J., T. O. Crist, J. M. Chase, M. Vellend, B. D. Inouye, A. L. Freestone, N. J. Sanders, H. V. Cornell, L. S. Comita, K. F. Davies, et al. (2011). Navigating the multiple meanings of β diversity: a roadmap for the practicing ecologist. Ecol. Lett. 14(1), 19–28.
  3. [3]Barwell, L. J., N. J. Isaac, and W. E. Kunin (2015). Measuring β-diversity with species abundance data. J. Anim. Ecol. 84(4), 1112–1122.DOI
  4. [4]Battiston, M., S. Favaro, D. M. Roy, and Y. W. Teh (2018). A characterization of product-form exchangeable feature probability functions. Ann. Appl. Probab. 28(3), 1423–1448.arxiv.org/abs/1607.02066
  5. [5]Broderick, T., J. Pitman, and M. Jordan (2013). Feature allocations, probability functions, and paintboxes. Bayesian Anal. 8(4), 801–836.arxiv.org/abs/1301.6647
  6. [6]Camerlenghi, F., S. Favaro, L. Masoero, and T. Broderick (2024). Scaled process priors for Bayesian nonparametric estimation of the unseen genetic variation. J. Am. Statist. Assoc. 119(545), 320–331.
  7. [7]Cao, Y., T. Xu, X. Yu, Z. Hua, H. Liu, W. Gu, H. Liu, H. Kang, Z. Wei, and Z. Xie (2024). Fungal aerosols over the Northwest Pacific to Arctic ocean: The influence of air mass and environmental factors. Atmos. Environ. 337, 120776.DOI
  8. [8]Carpenter, B., A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell (2017). Stan: A probabilistic programming language. J. Statist. Softw. 76, 1–32.
  9. [9]Colombi, A., R. Argiento, F. Camerlenghi, and L. Paci (2025). How many unseen species are in multiple areas? arXiv preprint arXiv:2502.04122.
  10. [10]De Blasi, P., S. Favaro, A. Lijoi, R. H. Mena, I. Prünster, and M. Ruggiero (2015, 2). Are gibbs-type priors the most natural generalization of the dirichlet process? IEEE Trans. Pattern Anal. Mach. Intell. 37, 212–229.
  11. [11]Doi, H., K. Fukaya, S. I. Oka, K. Sato, M. Kondoh, and M. Miya (2019). Evaluation of detection probabilities at the water-filtering and initial pcr steps in environmental DNA metabarcoding using a multispecies site occupancy model. Sci. Rep. 9(1), 3581.DOI
  12. [12]Franzolini, B., A. Lijoi, I. Prünster, and G. Rebaudo (2025). Multivariate species sampling models. arXiv preprint arXiv:2503.24004.arxiv.org/abs/2503.24004
  13. [13]Ghilotti, L., F. Camerlenghi, and T. Rigon (2026). Bayesian analysis of product feature allocation models. J. R. Statist. Soc. B 88(2), 540–566.
  14. [14]Ghilotti, L., F. Camerlenghi, T. Rigon, and M. Guindani (2025). Bayesian nonparametric modeling of multivariate count data with an unknown number of traits. arXiv preprint arXiv:2510.24526.arxiv.org/abs/2510.24526
  15. [15]Gotelli, N. J. and R. K. Colwell (2001). Quantifying biodiversity: procedures and pitfalls in the measurement and comparison of species richness. Ecol. Lett. 4(4), 379–391.DOI
  16. [16]Griffiths, T. L. and Z. Ghahramani (2011). The Indian buffet process: An introduction and review. J. Mach. Learn. Res. 12(4).DOI
  17. [17]Hess, O. C., T. van der Deure, M. Bolander, C. A. Leal Dutra, and J. Z. Shik (2025). The evolution of thermal performance curves in fungi farmed by attine ant mutualists in above-ground or below-ground microclimates. J. Evol. Biol. 38(1), 83–93.DOI
  18. [19]James, I. R. (1972). Products of independent beta variables with application to Connor and Mosimann’s generalized Dirichlet distribution. J. Am. Statist. Assoc. 67(340), 910–912.DOI
  19. [20]James, L. F. (2017). Bayesian Poisson calculus for latent feature modeling via generalized Indian buffet process priors. Ann. Statist. 45(5), 2016–2045.DOI
  20. [21]Lijoi, A., R. H. Mena, and I. Prünster (2007). Bayesian nonparametric estimation of the probability of discovering new species. Biometrika 94, 769–786.DOI
  21. [22]MacKenzie, D. I., J. D. Nichols, G. B. Lachman, S. Droege, J. Andrew Royle, and C. A. Langtimm (2002). Estimating site occupancy rates when detection probabilities are less than one. Ecology 83(8), 2248–2255.
  22. [24]Mauri, L. and D. B. Dunson (2025). Factor pretraining in Bayesian multivariate logistic models. Biometrika 112(4), asaf056.DOI
  23. [25]Ochiai, A. (1957). Zoogeographical studies on the soleoid fishes found in Japan and its neighbouring regions—II. Bull. Jpn. Soc. Sci. Fish. 22(9), 526–530. In Japanese.DOI
  24. [26]Ovaskainen, O., N. Abrego, B. Furneaux, B. Hardwick, P. Somervuo, I. Palorinne, N. R. Andrew, U. V. Babiy, T. Bao, G. Bazzano, et al. (2024). Global spore sampling project: A global, standardized dataset of airborne fungal DNA. Sci. Data 11(1), 561.DOI
  25. [27]Ovaskainen, O., N. Abrego, P. Halme, and D. B. Dunson (2016). Using latent variable models to identify large networks of species-to-species associations at different spatial scales. Methods Ecol. Evol. 7(5), 549–555.DOI
  26. [28]Pakpour, S., D.-W. Li, and J. Klironomos (2015). Relationships of fungal spore concentrations in the air and meteorological factors. Fungal Ecol. 13, 130–134.DOI
  27. [29]Pollock, K. H., J. D. Nichols, C. Brownie, and J. E. Hines (1990). Statistical inference for capture-recapture experiments. Wildl. Monogr., 3–97.
  28. [30]Rigon, T., C.-L. Hsu, and D. B. Dunson (2025). A Bayesian theory for estimation of biodiversity. arXiv:2502.01333.arxiv.org/abs/2502.01333
  29. [31]Royle, J. A., J. D. Nichols, M. Kéry, and E. Ranta (2005). Modelling occurrence and abundance of species when detection is imperfect. Oikos 110, 353–359.DOI
  30. [32]Schlegel, M., A. D. Treindl, J. Panziera, V. Zengerer, D. Zani, J. Brännhage, and A. Gross (2024). A case study on the application of spore sampling for the monitoring of macrofungi. Mol. Ecol. Resour. 24(4), e13941.DOI
  31. [33]Shen, Y., L. Masoero, J. G. Schraiber, and T. Broderick (2025). Double trouble: Predicting new variant counts across two heterogeneous populations. Bayesian Anal. 1(1), 1–33.arxiv.org/abs/2403.02154
  32. [34]Somervuo, P., S. Koskela, J. Pennanen, R. Henrik Nilsson, and O. Ovaskainen (2016). Unbiased probabilistic taxonomic classification for DNA barcoding. Bioinformatics 32(19), 2920–2927.DOI
  33. [35]Stolf, F. and D. B. Dunson (2025). Infinite joint species distribution models. Biometrika 112(4), asaf055.arxiv.org/abs/2402.13384
  34. [36]Tedersoo, L., M. Bahram, S. Põlme, U. Kõljalg, N. S. Yorou, R. Wijesundera, L. V. Ruiz, A. M. Vasco-Palacios, P. Q. Thu, A. Suija, et al. (2014). Global diversity and geography of soil fungi. Science 346(6213), 1256688.DOI
  35. [37]Teh, Y. and D. Gorur (2009). Indian buffet processes with power-law behavior. In Advances in Neural Information Processing Systems, Volume 22.
  36. [38]Tobler, M. W., M. Kéry, F. K. Hui, G. Guillera-Arroita, P. Knaus, and T. Sattler (2019). Joint species distribution models with species correlations and imperfect detection. Ecology 100(8), e02754.DOI
  37. [39]Tuomisto, H. (2010). A diversity of beta diversities: straightening up a concept gone awry. Part 1. Defining beta diversity as a function of alpha and gamma diversity. Ecography 33(1), 2–22.DOI
  38. [40]Wenger, S. J. and M. C. Freeman (2008). Estimating species occurrence, abundance, and detection probability using zero-inflated distributions. Ecology 89, 2953–2959.pubmed.ncbi.nlm.nih.gov/18959332
  39. [41]Whittaker, R. H. (1960). Vegetation of the Siskiyou Mountains, Oregon and California. Ecol. Monogr. 30(3), 279–338.DOI
  40. [42]Zito, A., T. Rigon, O. Ovaskainen, and D. B. Dunson (2023). Bayesian modeling of sequential discoveries. J. Am. Statist. Assoc. 118(544), 2521–2532.DOI

Paper details

vlnc:W_qnynet34
Version
v2 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents