WorksAgricultural and Biological Sciences
BIOTRAM: Biochemistry-informed Transformer for Relative Abundance via Metabarcoding
Abstract
Arthropod monitoring at scale begins with metabarcoding of bulk samples: trap collections of hundreds to thousands of specimens processed as one unit, extraction of DNA from the bulked specimens, then sequencing of the DNA from all specimens together. Metabarcoding reveals which taxa such a sample contains, but due to factors such as differing DNA amplification rates and body sizes, its read counts do not neatly correspond to how abundant the different taxa are relative to one another. This lack of relative abundance limits the data's utility for the multitude of ecological analyses that depend on it. We propose BIOTRAM: a neural network that estimates per-taxon specimen counts by dividing the observed read counts by a learned read yield per specimen, which it factors into the DNA each specimen contributes and how efficiently that DNA is amplified. On a paired dataset of 1,188 bottles (800 k specimens) processed both by metabarcoding and by comprehensive individual barcoding for ground-truthing, BIOTRAM reduces the Jensen-Shannon divergence between the estimated and the true composition from 0.3586 for raw read proportions (the usual proxy for abundance) to 0.1200. Trained on the same objective and inputs as four generic deep architectures, all with tuned loss weights, it reaches a count error 3.2% below that of the strongest of them and the lowest composition divergence. It also generalizes to out-of-domain data better, beating all baselines on bottles from unseen regions, later collection dates and unseen sequencing batches, and transferring to data from an unseen laboratory. Structuring the mapping from reads to counts around the steps of the experimental pipeline thus yields more accurate abundance estimates than leaving that mapping unconstrained.
Introduction
Insects are by far the most diverse multicellular taxon on Earth [7] and are fundamentally important for human existence given their foundational role in ecosystem processes [58]. Yet, they are undergoing extensive global declines [51, 19]; efforts to quantify this are limited by the paucity of reliable data [10]. Biodiversity indices used to track populations rely on estimates of both species richness (number of species) and abundance (number of specimens per species) [19, 50], while methods that provide only species lists but not abundances are of limited use. Malaise traps—tent-like structures that passively capture flying insects and direct them into a bottle of preservative—are widely used for capturing insects for study [25]. Although identification of their contents by morphology was common in the past, many species cannot be differentiated by morphology alone, and identifying species by manual or visual inspection is highly time consuming, expensive, and requires significant taxonomic expertise. Consequently DNA-based identification systems are increasingly used to identify captured species. Identifying specimens individually using DNA barcoding provides high-quality estimates of both richness and relative abundance, but is still relatively tedious and costly. Metabarcoding, a method wherein each bulk sample is processed as a unit by extracting and sequencing DNA from the entire collection, scales far better than DNA barcoding but does not produce individual specimen labels—only the number of amplicon reads—and so cannot directly estimate abundance [59, 23] (see Figure 1).
Figure 1. Individual barcoding vs. metabarcoding for estimating species abundance. Individual barcoding (manual sorting and per-specimen sequencing) gives accurate species counts but is slow and costly; metabarcoding (bulk amplification of the sample’s DNA) is fast and cheaper, but its read counts are not linearly related to specimen abundances.
Rather than abundance, the output of metabarcoding is a vector of sequence read counts, one count per taxon. Besides the number of individuals, four key factors influence how many reads each taxon receives: (1) how much DNA each individual contributes to the bulk DNA mixture, (2) how efficiently its barcode sequence is amplified during PCR, (3) how long the PCR reaction runs before reagents are exhausted, and (4) what fraction of the amplified product is sequenced. Factors 3 & 4 vary by protocol, and factors 1 & 2 vary across taxa by orders of magnitude, so read counts track specimen counts only loosely. In bottles whose specimens were also barcoded one by one, raw read counts, taken as relative abundances, are on average further from the true composition than a guess that every taxon is equally abundant (§ 5).
Machine learning is changing how metabarcoded reads are interpreted: foundation models trained on barcodes support identification and clustering even for unseen taxa [33, 15, 43]. Abundance has resisted the same treatment: identification models learn from reference libraries of millions of individually barcoded specimens (deWaard et al., 2019; Gharanee et al., 2024), but abundance requires bulk samples whose composition is known specimen by specimen—the tedious, costly work that metabarcoding bypasses. The reads-to-specimens mapping also differs by taxon, and per-taxon correction factors [32] cannot be calibrated for taxa that are rare or absent from calibration samples.
BIOTRAM, a biochemistry-informed transformer for relative abundance via metabarcoding, is instead structured around the steps of the experimental pipeline: it estimates specimen counts by dividing each taxon’s reads by a learned read yield per specimen, factored into per-specimen DNA contribution and amplification efficiency, so it can produce estimates for taxa absent from training. Its taxonomic unit is the Barcode Index Number (BIN), a cluster of near-identical barcodes proxying for species that usually agrees with species assigned by morphology [41]. Our contributions are as follows:
A paired DNA barcoding-metabarcoding dataset (§ 3). 1,188 bottles processed with both metabarcoding and (for ground-truthing) individual barcoding, yielding 414,640 matched (reads, count) pairs across 132,184 unique BINs and 578 sites—more than an order of magnitude larger than existing data.
A bespoke architecture built on biological and chemical knowledge (§ 4.3). Rather than a naïve model, we propose a staged model based on the biochemical properties of the metabarcoding process.
Latent predictions of primer binding efficiency and body size (§ 4.1). Our model produces physically interpretable latent estimates of both properties, which directly shape the relationship between specimen counts and read counts per BIN and bottle.
Controlled evaluation under distribution shift (§ 5). We compare BIOTRAM against deep baselines trained with the same objective and inputs, generalizing to novel collection locations (spatial blocking), novel time-points (temporal cut-off), and across metabarcode sequencing methods (data from another laboratory).
Related work
Metabarcoding studies have long sought relative abundances from bulk samples, mostly with regression or other statistical models ([23, 48, 28]). For Malaise traps, Furneaux et al. [14] analyzed a paired barcode-metabarcode dataset of 45 bulk samples from Sweden and Finland with mixed models, and found strong correspondence between read counts and abundance; they also modeled which BINs are present as individuals, even when undetected by metabarcoding, and how BIN-specific abundances affect detection. Iwaszkiewicz-Eggebrecht et al. [22] applied a hierarchical Bayesian model that incorporates spike-in data to 15 Swedish samples with comprehensive barcode and metabarcode data. Both studies found close correspondence between read counts and relative abundances from barcoding, but were fit to a few dozen samples from northern Europe, vs. 1,188 bottles from 578 global sites here. We refit the hurdle mixed model of Furneaux et al. [14] to our data as a baseline (§ 5 and Appendix § 18).
Metabarcoding studies of other taxa also estimate relative abundance, including that of plants ([31]), freshwater macroinvertebrates ([11]), and fishes ([29]). Metagenomic workflows such as SPIKEPIPE ([24]) avoid PCR bias but cost far more than metabarcoding. To our knowledge, none of these methods uses a neural network trained on paired data at this scale.
Dataset
Training requires metabarcoding data paired with ground-truth specimen counts—these come from separating and barcoding every specimen in a bottle, which is an expensive process. See Figure 1 for an overview of the respective individual barcoding and metabarcoding data pipelines. We assembled such data for Malaise-trap bottles from the LIFEPLAN project (deWaard et al., 2024; Hardwick et al., 2024) and the Global Malaise Program ([17]), as described below. Permissions to use each bottle were arranged with the data owners (Appendix § 10).
Barcoding. DNA was extracted from every bottle by mild lysis, a chemical extraction that draws DNA from the specimens while leaving them intact, and every specimen was then barcoded individually. Metabarcoding amplified a ~418 bp stretch of the cytochrome oxidase subunit I gene (COI; BF3/BR2 primers; [12, 13]), in three PCR replicates, which were sequenced on an Illumina NextSeq 2000 and processed with the Metabarcoding Analysis Pipeline (MAP; Prosser et al. [39]). MAP clustered the reads into operational taxonomic units (OTUs), each matched to a BIN in a reference library (Appendix § 11). OTUs with no BIN match within 2.3% sequence identity were assigned to their nearest BIN, because new BINs can only be created within BOLD. Individual barcoding yields a per-BIN specimen count for every bottle, which BIOTRAM learns to predict from the metabarcoding data alone. Individual barcoding fails for under 10% of specimens (Orsholm et al. [34]), so some BINs that metabarcoding detects but barcoding misses are present, not false detections.
At deployment, a monitoring program has only the metabarcoding results, without individual sorting or prior calibration. We therefore drop the BINs detected by individual barcoding but absent from metabarcoding, and keep the true zeros of BINs seen only in metabarcoding, which the model is trained and evaluated to predict.
The paired dataset spans 1,188 bottles from 578 sites collected between 2014 and 2025. It contains 414,640 rows, one per BIN per bottle, across 132,184 BINs and 800,875 counted specimens. A typical bottle holds 288 BINs and 375 specimens (medians; maxima 1,534 and 16,900), and the dataset covers 1,283 families.
As our goal is to predict the distribution of species within a bottle, the bottle is the model’s unit of processing, and we partition the data by bottle. Specimens are almost entirely arthropods (Insecta 73.7%, Collembola 21.1%, Arachnida 5.1%). Most BINs lack genus and species labels, so we use BINs instead of named species.
Reads vs. counts. We call the specimen count of BIN in bottle and its relative abundance; the vector is the bottle’s composition. Read shares are the usual proxy for composition, but they are biased [32, 14, 22], and in our data they track it poorly: on a scale, read counts explain 23% of the variance in specimen counts, rank BINs within a bottle at a mean Spearman correlation of 0.49, and separate present BINs from false detections at an AUC of 0.78. The most-sequenced BIN of a bottle is also its most abundant in only 13.8% of bottles.
Excess specimens. In 603 of the 1,188 bottles, technicians counted at least one highly abundant morphospecies by eye instead of barcoding each specimen (Figure 3), so these bottles lack complete per-BIN ground truth. We evaluate only on the 585 bottles without this excess and use 455 of the others, those least affected by it, for training only (Appendix § 14).
Figure 3. DNA barcoding data pipeline. Each bottle is collected from a Malaise trap and undergoes mild lysis to release DNA into solution while preserving the physical specimens. From this lysate, a DNA sample is drawn for metabarcoding: three independent PCR replicates are amplified from the bulk sample and passed through the Metabarcoding Analysis Pipeline (MAP) to produce a single bulk metabarcode. In parallel, the physical sample is fully sorted and its specimens are plated onto 96-well microplates for individual barcoding, yielding up to 95 individual barcodes per plate (one well is left empty as a negative control); where many specimens appear to belong to the same species, only some representatives are barcoded, while the remaining specimens are counted and set aside as excess. Individual barcode and metabarcode records are then joined by Barcode Index Number (BIN).
Methods
We assume that the read count of BIN in bottle for replicate is, up to a bottle-level constant that absorbs the replicate’s total read count, the product of three quantities: (1) the specimen count , which we predict; (2) the amplifiable DNA per specimen, which on average scales with the BIN’s expected body mass, the body-mass term ; and (3) the log amplification of its barcode in replicate before the reagents ran out:
See Appendix § 12 for our derivation of this forward relationship.
Body size and primer binding efficiency
Two per-BIN inputs inform the terms of Eq. (1) and can be estimated from data outside metabarcoding: body size, which the body-mass term should track, and primer binding efficiency, which shapes the log amplification . We estimate both from public specimen images and reference sequences, freeze the estimators, and give their outputs to BIOTRAM and every baseline. § 5 evaluates these models and Appendices § 13 and § 15 detail them.
Body size. Larger individuals have more surface in contact with the DNA extraction fluid, so more of their DNA enters the amplification process. To estimate specimen sizes, we used the size of photographed insects as a proxy (Figure 2B). From the public specimen images in the Barcode of Life Data System (BOLD), we assembled 639,751 images associated with BIN labels and segmented each image using pretrained vision models (Appendix § 15). The pipeline segments the whole organism and then carves away spread wings, which add projected area but little tissue (Figure 5). For each BIN, we averaged the projected body area over specimens, giving an expected size for 78,259 BINs, representing 50.2% of our dataset’s BINs. To generalize this model beyond BINs with imaged examples, we fine-tuned a pretrained barcode model, BarcodeBERT (Millan Arias et al., [33]), to estimate the mean log area of a BIN and its within-BIN spread from a single barcode within the BIN; BIOTRAM reads these predicted values for every BIN.
Figure 2. BIOTRAM method overview. (A) Binding Rate Estimator predicts each primer’s binding rate from its barcode, trained against rates estimated for longer sequences with known DNA at primer sites. (B) Insect Size Estimator predicts a BIN’s average specimen size from a representative barcode sequence, trained on estimates found from paired image data. (C) BIOTRAM’s architecture integrates barcode, taxonomic, and sample metadata to jointly predict each BIN’s body-mass term and presence; an attention module corrects raw read counts for amplification bias across replicates before the corrected estimates are merged into a final abundance prediction, trained with negative binomial, Jensen–Shannon divergence (JSD), binary cross-entropy (BCE) and Huber losses (§ 4.4). For submodule component design, see Figure 6.
Figure 5. Segmentation and the wing carve, on two winged specimens. Each row is one specimen: the photograph, the envelope from the two full-organism models (SAM-HQ and SAM 3), and the carved body the pipeline keeps. Masks are drawn translucently over the photograph so the specimen stays visible underneath. LPGEQ49117-24 (Lepidoptera) goes from 222.1 to 12.6 mm, 94% of the silhouette removed; CRBCA996-24 (Plecoptera) from 14.5 to 2.4 mm, 83% removed. Areas are the calibrated values the size anchor is built from.
Primer binding efficiency. The PCR process consists of copying the barcode region between two primers, short fixed sequences that bind to the DNA on either side of it (Figure 2A). The sequence at these binding sites varies across species, and a primer binds better the more closely the site matches it. These sites are not shown in reads, because amplification overwrites them with the primer sequences, so we take them from two external sources: full-length barcode records, which extend beyond the amplicon and give the forward site for 91.5% of BINs, and mitogenome records, which give both the forward and the reverse sites for another 1.2%. We score each retrieved 20-nucleotide site to estimate a binding efficiency between 0 and 1; for the remaining BINs, we predict the efficiency from the amplicon with a ridge and kernel ridge regression (Appendix § 13).
Sources of variation in reads per specimen
We quantify what distorts the relationship between reads and specimen counts: how much is attributable to the BIN itself (e.g., primer binding, body size), the reaction as a whole, and detection noise. Each source of variation motivates a specific component of the model.
Most of the variation is per-BIN, but it cannot be looked up. The number of reads one specimen produces spans 4.2 orders of magnitude across BINs, and BIN identity explains 80.4% of that variation. The classical remedy corrects each BIN by its own properties (McLaren et al., [32]), a structure our model keeps, but 60% of BINs appear in a single bottle and 19% of test rows carry a BIN absent from training, which a lookup table cannot handle. We, instead, compute the per-BIN quantities from the barcode sequence, since primer binding and body size are conserved among relatives.
False detections. 50.9% of the BINs detected in a bottle never occur: contamination, cross-talk between bottles sequenced together, traces of environmental DNA and clustering artifacts all leave reads where no specimen was present (Schnell et al., [47]; Zinger et al., [60]). A dedicated presence head, trained with its own loss, gates the abundance estimate, so a detected BIN can be reported absent.
Replicate disagreement. Only 46.8% of detections appear in all three PCR replicates and 29.4% in just one. When little of a BIN’s DNA is present, whether it amplifies is partly chance, so replicate disagreement is itself evidence of how much DNA was there. The model takes the three replicate counts as inputs, not their sum.
Bottle-level saturation. A PCR amplifies exponentially until its fixed reagent budget is spent, then plateaus. How long that takes depends on the whole bottle. Because BINs amplify at different rates, the ratio between two BINs’ reads depends on every BIN in the bottle (Suzuki & Giovannoni, [49]): the emptiest quarter of bottles yields a median of 5,896 reads per specimen, roughly 8× the 720 of the fullest quarter. Sequencing then samples only a fraction of the amplified product. Both effects require bottle-level terms.
The model
BIOTRAM predicts and , together with the probability that the BIN is present at all (), and inverts Eq. (1) against the observed read count,
with the replicate’s total reads, the bottle’s total specimen count, the logit that BIN is truly present, the sigmoid function, and a small learned network that merges the three replicate channels, starting from their mean. The channels are each BIN’s smallest, median and largest read count, so they do not depend on the arbitrary order of the replicates.
Two trained networks predict the per-BIN quantities and . Each computes a frozen BarcodeMamba embedding of the barcode (Gao & Taylor, [15]; Gao et al., [16]) through its own attention layer, adds a learned taxonomy embedding and the bottle’s location and collection date (through a pretrained geographic encoder (Klemmer et al., [27]) and a seasonal encoding, respectively) and passes the result through an MLP. The frozen embedding is what lets an unseen BIN inherit from its relatives. The body mass network also reads the barcode-predicted body size of § 4.1, the presence network reads the primer efficiencies, the bottle’s total specimen count and the BIN’s replicate read counts, so is a read-informed presence probability.
The amplification term comes from a self-attention module over every BIN of the bottle, run once per replicate channel. Each BIN is characterized by its barcode, its two primer efficiencies (each with a binary input marking whether it was computed from a retrieved binding site or predicted; § 4.1), its predicted body size, and its read count and read share within the bottle. Every BIN also receives a summary of the bottle: the mean of all its BINs’ representations, weighted by a softmax over their amplification demand, estimated from the predicted body size and the two primer efficiencies with a learned temperature, along with the bottle’s covariates and the replicate’s total reads. The BINs expected to consume most of the shared reagents thus dominate the context from which every amplification term is estimated. A read-blind variant estimates without the BIN’s own read (Appendix § 18), and Appendix § 25 measures what each input contributes. The count is read out as the mean of a negative binomial whose dispersion is predicted per row. Apart from the replicate merge, the arithmetic that turns the predicted quantities into a count is fixed by Eq. (2) (overview in Figure 2C, full architecture in Figure 6).
Figure 6. Full BIOTRAM architecture. Extends Figure 2C. A frozen barcode embedding, a learned taxonomy embedding and the bottle’s location and date feed two per-BIN heads (the body-mass term and the presence logit ) and a masked-attention module that runs over every BIN of the bottle, once per replicate, and emits the amplification term from the BIN’s identity, its primer and size features, its own read count and the replicate’s total read count. Equation (2) then combines these quantities with the raw measured reads; the three per-replicate estimates are merged and gated by . A separate MLP predicts the negative-binomial dispersion from , and .
Training objective
Let be the bottles in a batch and the pairs of the whole batch, and write for the observed count, for Eq. (2)’s prediction, and for the predicted presence probability. Training minimizes a negative-binomial count likelihood (NB), which accommodates the over-dispersion of ecological counts, whose variance exceeds their mean, plus a per-bottle divergence on composition, a presence loss, and a term tying the predicted reads per specimen to its observed value:
The NB variance keeps a taxon whose count genuinely varies between bottles from having that variance absorbed as bias in the mean. scores the within-bottle composition, with at ; it is also the reported metric and the checkpoint-selection criterion. We use JSD rather than KL because it is symmetric – over-predicting a rare BIN and under-predicting an abundant one are penalized alike — and because it stays finite on the 50.9% of rows whose truth is zero, which , weighted by the true abundance, does not directly penalize. trains the presence probability of Eq. (2) directly on the presence label , so that a BIN with reads but no individuals is treated as a false detection rather than a small count. Unlike the other terms, supervises an intermediate quantity rather than the output. By Eq. (1), is the log number of reads per specimen of BIN , where is the mean of over the three replicates. Its target is the mean of over the training bottles, with summed over the replicates, and is the standard deviation of that quantity across those bottles. Only BINs observed in at least three training bottles contribute to the loss (, otherwise ), and is the -weighted mean of over the batch. is the Huber loss with threshold . Training batches are whole bottles (Appendix § 16).
Results
We evaluate BIOTRAM against a broad range of baselines, on random splits, under three kinds of distribution shift and on an external dataset with a different laboratory workflow.
Evaluation design and metrics. Data is partitioned at the bottle level. The 585 excess-free bottles are split 409 / 87 / 89 into train/val/test, and 455 of the 603 bottles with excess specimens (Appendix § 14) join the training set only, for 864 training bottles in all. The baselines span linear models, nearest neighbours, tree-based models, zero-inflated and hurdle models, deep set and attention-based architectures. All receive the same inputs and are each tuned on their own hyperparameter search space against Jensen–Shannon divergence (JSD) between the predicted and the true within-bottle composition on the validation set, our primary metric. From the same predicted counts, we also score the mean absolute error (MAE; per bottle, on rows with no individuals, , on the rest, , and on log counts, MAE-log), presence AUC, Spearman correlation and two summary statistics used frequently in ecology (Appendix § 17). Two models are level when a two-sided Welch test over seeds cannot separate them.
Every deep model, BIOTRAM included, is trained on the objective of Eq. (3), with its three auxiliary weights tuned for it under one protocol and budget (Appendix § 24). Its term supervises the log reads per specimen, which by Eq. (1) is . A baseline has no amplification term, but its predicted count gives the same quantity as , so the term applies to every model.
Overall accuracy. Table 1 reports accuracy on the held-out test bottles. Among the deep models trained on our data, BIOTRAM is the most accurate on both the counts and the composition. Its mean absolute error per bottle is 0.699, which is 3.2% lower than that of the strongest baseline (0.723), and its JSD is 0.1200, against 0.1234 for the best baseline. Both differences are significant against each of the four baselines (two-sided Welch test over seeds). The first row of the table is the hurdle mixed model of Furneaux et al. [14], the established statistical approach to this problem. It is fitted on its own likelihood rather than on Eq. (3), so we show it for reference, outside the matched comparison. Simple predictors do much worse (Table 4). A uniform composition, with every taxon equally abundant, scores 0.3174 on JSD and the raw read shares score 0.3586. The classical per-BIN correction of McLaren et al. [32] reaches only 0.3435 JSD, because 60% of BINs appear in a single bottle and most test rows therefore have no correction to apply.
| Model | JSD ↓ | Spearman ↑ | Pres. AUC ↑ | MAE/bottle ↓ | MAE ↓ | MAE ↓ |
| Hurdle GLMM (Furneaux et al., [14]) | 0.1947 | 0.609 | 0.8419 | 1.498 | 0.730 | 2.286 |
| Transformer @ 9.6M | 0.1257 ± 0.0019 | 0.741 ± 0.003 | 0.9156 ± 0.0017 | 0.723 ± 0.020 | 0.295 ± 0.023 | 1.395 ± 0.048 |
| MLP @ 11.1M | 0.1234 ± 0.0006 | 0.741 ± 0.001 | 0.9154 ± 0.0007 | 0.737 ± 0.025 | 0.303 ± 0.013 | 1.409 ± 0.047 |
| DeepSets @ 10.1M | 0.1256 ± 0.0010 | 0.742 ± 0.002 | 0.9162 ± 0.0013 | 0.752 ± 0.036 | 0.325 ± 0.020 | 1.453 ± 0.110 |
| 1-D CNN @ 11.5M | 0.1235 ± 0.0008 | 0.737 ± 0.001 | 0.9145 ± 0.0008 | 0.762 ± 0.016 | 0.321 ± 0.010 | 1.473 ± 0.036 |
| BIOTRAM | 0.1200 ± 0.0009 | 0.741 ± 0.001 | 0.9136 ± 0.0005 | 0.699 ± 0.017 | 0.279 ± 0.016 | 1.341 ± 0.024 |
| TabICLv2 (in-context, 28.5M) | 0.1238 ± 0.0001 | 0.751 ± 0.001 | 0.9239 ± 0.0005 | 0.704 ± 0.003 | 0.249 ± 0.002 | 1.413 ± 0.006 |
Table 1. In-domain test performance. Test split of the excess-free bottles; every model sees the same inputs. The hurdle mixed model of Furneaux et al. [14] is fitted on its own likelihood with its published predictors (Appendix § 18); the deep models are trained on the objective of Eq. (3), each with its auxiliary weights tuned for it (Appendix § 24). JSD scores the predicted composition, Spearman the ranking of the counts over all test rows, presence AUC how well the predicted counts separate present from absent BINs, and MAE/bottle the error on the counts within each bottle, averaged over bottles; MAE and MAE are the count error over all BINs truly absent or present. Values are means over seeds ± one standard deviation across seeds (the hurdle mixed model is a single fit). We highlight the best and <u>second best</u> model.
| Model | JSD ↓ | MAE/bot. ↓ | MAE-log ↓ | Pres. AUC ↑ | Spearman ↑ | Shannon ↑ | Mantel ↑ |
| Reference | |||||||
| Every taxon equally abundant | 0.3174 | 1.22 | 0.502 | 0.500 | – | 0.627 | 0.724 |
| Raw relative reads | 0.3586 ± 0.0000 | 2.00 | 0.424 | 0.781 | 0.486 | 0.369 | 0.635 |
| Per-BIN offset (McLaren et al., [32]) | 0.3435 ± 0.0000 | 2.28 | 0.424 ± 0.000 | 0.803 | 0.519 | 0.443 | 0.588 |
| Linear models | |||||||
| Ridge regression | 0.2159 | 1.25 | 0.424 ± 0.000 | 0.802 | 0.539 | 0.622 | 0.776 |
| Mixed models | |||||||
| Hurdle GLMM (Furneaux et al., [14]) | 0.1947 | 1.50 | 0.430 | 0.842 | 0.583 | 0.731 | 0.830 |
| Nearest neighbours | |||||||
| -nearest neighbours | 0.1721 ± 0.0029 | 0.98 ± 0.01 | 0.319 ± 0.002 | 0.869 ± 0.001 | 0.626 ± 0.003 | 0.777 ± 0.006 | 0.839 ± 0.007 |
| Tree-based models | |||||||
| Random forest | 0.1517 ± 0.0001 | 0.92 ± 0.00 | 0.288 ± 0.000 | 0.900 ± 0.000 | 0.669 ± 0.000 | 0.765 ± 0.001 | 0.854 ± 0.003 |
| Gradient boosting | 0.1653 ± 0.0040 | 0.92 ± 0.02 | 0.296 ± 0.002 | 0.884 ± 0.006 | 0.641 ± 0.011 | 0.751 ± 0.021 | 0.841 ± 0.005 |
| Zero-inflated models | |||||||
| Zero-inflated ridge | 0.1742 ± 0.0000 | 0.95 ± 0.00 | 0.304 ± 0.000 | 0.889 | 0.646 | 0.680 ± 0.000 | 0.814 ± 0.000 |
| Two-stage hurdle | 0.1472 ± 0.0013 | 0.93 ± 0.00 | 0.290 ± 0.001 | 0.906 ± 0.000 | 0.678 ± 0.001 | 0.749 ± 0.004 | 0.852 ± 0.003 |
| Deep models | |||||||
| MLP | 0.1247 ± 0.0016 | 0.74 ± 0.00 | 0.245 ± 0.003 | 0.916 ± 0.000 | 0.686 ± 0.002 | 0.850 ± 0.011 | 0.894 ± 0.005 |
| DeepSets | 0.1261 ± 0.0013 | 0.74 ± 0.01 | 0.244 ± 0.001 | 0.913 ± 0.000 | 0.686 ± 0.001 | 0.881 ± 0.007 | 0.876 ± 0.001 |
| 1-D CNN on the barcode | 0.1250 ± 0.0010 | 0.73 ± 0.02 | 0.230 ± 0.004 | 0.916 ± 0.002 | 0.686 ± 0.003 | 0.865 ± 0.012 | 0.895 ± 0.006 |
| Transformer over the bottle | 0.1256 ± 0.0008 | 0.71 ± 0.01 | 0.229 ± 0.003 | 0.913 ± 0.002 | 0.682 ± 0.002 | 0.886 ± 0.007 | 0.889 ± 0.008 |
| BIOTRAM | 0.1194 ± 0.0007 | 0.70 ± 0.01 | 0.232 ± 0.002 | 0.914 ± 0.001 | 0.688 ± 0.001 | 0.894 ± 0.007 | 0.890 ± 0.006 |
| BIOTRAM, read-blind ablation (no BIN’s own read in the amplification module) | |||||||
| BIOTRAM, read-blind (ablation) | 0.1229 ± 0.0012 | 0.76 ± 0.02 | 0.242 ± 0.002 | 0.913 ± 0.000 | 0.686 ± 0.001 | 0.879 ± 0.007 | 0.886 ± 0.005 |
| TabICLv2 (in-context, 28.5M) | 0.1238 ± 0.0001 | 0.70 ± 0.00 | 0.226 ± 0.001 | 0.924 ± 0.000 | 0.699 ± 0.001 | 0.820 ± 0.005 | 0.885 ± 0.002 |
Table 4. Test set, occurrence-count target. Mean over seeds ± one standard deviation across seeds. Bold is the best value in each column and underline the second-best value. All models see the same rows, the same split and the same evaluation code, and are selected on validation JSD. Every deep model is at the 10.5M rung of the capacity ladder, BIOTRAM’s own size.
The ranking metrics show a different pattern. On rank correlation, BIOTRAM is ahead of one of the baselines and level with three, and on presence AUC it trails four of them. This follows from the mechanism. Dividing the measured reads by an estimated amplification and body mass (Eq. (2)) constrains the scale of each prediction, which is what MAE measures, but it offers no comparable help in deciding which of two taxa in a bottle is more abundant. That ranking is largely decided by presence, which the read counts already carry and which an unconstrained network can fit directly.
TabICLv2 (Qu et al., [40]) was kept apart since it works differently. It is pretrained on synthetic tabular datasets and is not trained on our data at all; instead, it predicts each test bottle by attending to examples from the training set. It has the best presence AUC, MAE-log and of any model. BIOTRAM is level with it on count accuracy (MAE 0.699 against 0.704) and ahead of it on composition, with 35% of its parameters. Attending to the training set also makes TabICLv2 expensive to run: it needs times more compute per bottle than BIOTRAM (Appendix § 18).
Error structure and distribution shift. By true abundance, BIOTRAM is most accurate on the rows with no individuals, ahead of four of the matched baselines, and on counts above 100, ahead of three of them and level with one: only 0.1% of rows, but they carry a large share of each bottle’s individuals. It is also ahead of four of them on rows whose BIN was seen in training, but behind at least one of them on three of the four buckets between 1 and 100, and behind two of them on BINs never seen in training (Appendix § 19). A random split shares sites, seasons and sequencing batches between training and testing, which deployment does not, so we also retrain with a region never seen, bottles collected later than any in training, or a new sequencing batch held out (Table 2). The temporal shift is the hardest, adding +0.0662 in JSD against +0.0419 for the spatial and +0.0517 for the batch shift. Against each regime’s best baseline (Table 2), BIOTRAM is ahead of it on the spatial split (−0.0090 JSD), ahead of it on the batch split (−0.0047) and ahead of it on the temporal one (−0.0099; Appendix § 21). On fifteen Malaise-trap bottles from Sweden (4,057 specimens, each barcoded individually), metabarcoded at another laboratory with its own protocol (unpublished data; Appendix § 22), BIOTRAM stays closer to the true composition than the read shares and is level with the Transformer and TabICLv2; the best baseline there is the hurdle mixed model of Furneaux et al. [14], closer on composition but with the largest count error of any model (Table 2). Beyond point estimates, BIOTRAM’s NB predictive distribution is conservative: its central 90% intervals cover 0.967 of held-out rows (Appendix § 20).
| Regime | Simulated domain change | BIOTRAM JSD ↓ | BIOTRAM Δ ↓ | BIOTRAM MAE/bottle ↓ | Best baseline Model | Best baseline JSD ↓ | Best baseline Δ ↓ | Best baseline MAE/bottle ↓ |
| Random | in-domain reference | 0.1194 | – | 0.700 | 1-D CNN | 0.1238 | – | 0.759 |
| Spatial | a region never sampled in training | 0.1612 | +0.0419 | 0.977 | Transformer | 0.1702 | +0.0445 | 1.076 |
| Lane | a new sequencing batch | 0.1711 | +0.0517 | 0.670 | Transformer | 0.1758 | +0.0501 | 0.848 |
| Temporal | bottles collected later than any in training | 0.1856 | +0.0662 | 0.896 | Transformer | 0.1954 | +0.0697 | 1.029 |
| Collaterals dataset | different location, protocol (lysis) | 0.2189 | +0.0995 | 2.781 | Hurdle GLMM | 0.2026 | +0.0078 | 4.146 |
| Collaterals dataset | different location, protocol (homogenate) | 0.2177 | +0.0984 | 2.544 | Hurdle GLMM | 0.1965 | +0.0018 | 3.981 |
Table 2. Out-of-domain test performance. Each regime holds out one stratum of bottles from training; the collaterals rows are the external dataset of Appendix § 22. BIOTRAM is compared with the best baseline of each row, among the four matched deep baselines and the hurdle mixed model (Appendix § 21); Δ is each model’s change in JSD from its own random-split score, MAE/bottle is the count error of Table 1, and the best value of each metric is in bold.
Robustness of the comparison. Table 1 rests on three choices for every deep model: loss weights tuned for it, a size close to BIOTRAM’s and the full training set. We vary each in turn to test whether BIOTRAM’s lead depends on them. With one set of loss weights shared by every model (Appendix § 24), only the architecture differs: BIOTRAM’s MAE then separates from four of the four deep baselines and its JSD from four, with an MAE 5.8% below the strongest of them, against 3.2% with tuned weights (Appendix § 18). Across a capacity ladder from 200k to 128M parameters, retuned at every rung, no model moves by more than 0.0062 JSD, and BIOTRAM at its worst size (0.121) is still ahead of the 1-D CNN at its best (0.123). Only training-set size reverses the lead: on nested subsets of the training bottles, BIOTRAM trails every other model up to 216 bottles and leads the four deep baselines from 648 on, and every model is still improving at the full set, BIOTRAM most steeply (Appendix § 26). Its structure thus pays off once there are enough bottles to learn the per-BIN quantities it relies on.
External anchors. Mean log area and mean log dry mass correlate at across the 587 weighed BINs, BarcodeBERT predicts the mean log area of held-out BINs from the barcode alone at (against 0.82 for the nearest training barcode), and correlates with a BIN’s empirical at . The reverse primer site comes only from mitogenomes (1.2% of BINs) and is otherwise imputed, which can be a concern when deploying the model on new communities (Appendix § 13.§ 2).
Ablations. A hierarchical ablation over 20 capability blocks, each removed from the reported configuration and compared seed-for-seed against a control, finds that the two largest effects are data and chemistry rather than architecture: the primer binding-efficiency block (+0.0127 test JSD) and the extra training bottles with excess (+0.0117). The amplification module’s cross-BIN attention costs +0.0054, and giving it the BIN’s own read is worth +0.0039 on its own (full results in Appendix § 25).
Do the predicted quantities mean what they are named? The model’s distinguishing property is its intermediate quantities, which should correspond to physical quantities and explain its predictions. Since the primer efficiency and body mass receive the corresponding external estimates as input features, correlating them with those estimates would be circular. Instead, we check, for BIOTRAM at the development weights of Table 3, how well the predicted log reads per specimen, averaged per BIN over the test bottles (; Eq. (1)), correlates with its empirical value: it reaches Pearson +0.627 ± 0.007 over 12 seeds, and the read-blind variant, whose amplification term never sees the read and so cannot reproduce it mechanically, still reaches +0.461 ± 0.013 (Appendix § 23).
| Model | Params | JSD ↓ | Spearman ↑ | MAE/bottle ↓ | MAE=0 ↓ | MAE>0 ↓ |
| Every taxon equally abundant | – | 0.3174 | – | 1.219 | 1.000 | 1.723 |
| Raw relative reads | – | 0.3586 | 0.504 | 2.003 | 0.544 | 3.633 |
| Hurdle GLMM ([14]) | – | 0.1947 | 0.609 | 1.498 | 0.730 | 2.286 |
| Transformer | 9.6M | |||||
| 1-D CNN | 11.5M | |||||
| MLP | 11.1M | |||||
| DeepSets | 10.1M | |||||
| BIOTRAM | 9.9M | |||||
| BIOTRAM, read-blind ablation | 9.9M | |||||
| TabICLv2 (in-context) | 28.5M |
Table 3. Architecture only: every deep model at the same loss weights. As Table 1, but every deep model is trained with the same loss weights, , rather than its own tuned ones, so that only the architecture differs. The read-blind row is BIOTRAM without the BIN’s own read in the amplification module (§ 4). The first two rows are the fixed reference predictors of Table 4, and Params is each model’s number of parameters. We highlight the best and second best model.
Discussion
We have presented a new dataset that pairs metabarcoding read counts with specimen counts, and designed BIOTRAM, a machine learning model built specifically to map between those two quantities, trained and evaluated on that dataset.
Limitations. The number of training bottles, not model capacity, limits accuracy: larger variants of BIOTRAM do not improve it, no baseline gains more than 0.0062 JSD across the capacity ladder, and every model is still improving at the full training set (Appendix § 26). The counts identify only the product of amplification and body mass, and the amplification module sees the BIN’s own read, so the likelihood alone pins down neither term. What separates them is what each component reads: of the two, only the amplification module sees the primer features, the reads and the rest of the bottle. The latent quantities are therefore cleanest to interpret in the read-blind variant (Table 3), at a cost in accuracy. Body size varies within a BIN with life stage, sex and season, the image-based estimate depends on specimen pose, and the weighed-specimen dataset is small. The largest counts carry the least certain training labels: excess-affected taxa are 3.2% of BINs but 40% of the rows with more than a hundred specimens, and in the bottles with excess the training labels count only the individually barcoded specimens, leaving the excess out (Appendix § 14). The deep models, BIOTRAM included, also receive the bottle’s total specimen count , here the sum of its barcoded counts, which is expected to be known at deployment. BIOTRAM does not yet use spike-ins, known quantities of specimens or DNA added to a sample, which improve quantification in metabarcoding and metagenomics ([6, 21, 24] (Iwaszkiewicz-Eggbrecht et al. 2026)). Finally, the pretrained geographic encoder ([27]) contributes little, suggesting that general-purpose location embeddings do not transfer readily to biodiversity prediction.
Conclusions. BIOTRAM treats the biochemistry of metabarcoding as part of the model. Every quantity it predicts has a physical meaning, two are anchored to independent measurements, and, apart from a learned merge of the three replicates, the arithmetic that converts them into counts is fixed. With every model’s loss weights tuned, it reaches the lowest composition divergence and the lowest count error of the architectures we compared. Every model we tested degrades substantially under spatial, temporal and batch shift. Primer binding efficiency is among the largest gains in our ablation, which argues for single-specimen barcodes that retain the primer binding sites (Appendix § 13).
Acknowledgements
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. GWT and SCL are supported by the INSPIRE (Integrated Network for the Surveillance of Pathogens: Increasing Resilience and capacity in Canada’s pandemic response) project funded through the Canada Biomedical Research Fund (grant CBRF2-2023-00008), the Biomedical Research Infrastructure Fund, and the Ontario Research Fund. This work was supported by the AI and Biodiversity Change (ABC) Global Center, which is funded by the US National Science Foundation under Award No. 2330423 Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada CIFAR AI Chairs program. GWT is also supported by the Canada Research Chairs program (grant CRC-2021-00561). This study was enabled by support to PDNH from the Government of Canada through Genome Canada and Ontario Genomics (OGI-208 and OGI-233), by the New Frontiers in Research Fund (NFRFT-2020-00073), and Canada First Research Excellence Fund (CFREF)–Food from Thought. Infrastructure required to carry out the work was acquired with support from the Canada Foundation for Innovation (CFI) and the Gordon and Betty Moore Foundation. A Major Science Infrastructure award (MSI 42450) from CFI plays a key role in sustaining the CBG’s informatics and sequencing platforms.
Author Contributions
Conceptualization: Emma Boehly, Scott C. Lowe, Ken A. Thompson, Will Armstrong, Graham W. Taylor.
Methodology: Emma Boehly, Scott C. Lowe, Ken A. Thompson, Anthony Fuller, Will Armstrong, Neelu Madan, Tiancheng Gao, Graham W. Taylor.
Software: Emma Boehly, Ken A. Thompson, Will Armstrong, Sameer Padhye.
Validation: Emma Boehly, Scott C. Lowe, Graham W. Taylor.
Formal analysis: Emma Boehly, Ken A. Thompson, Will Armstrong.
Investigation: Emma Boehly, Scott C. Lowe, Ken A. Thompson, Will Armstrong, Sameer Padhye, Jayme Sones, Kate H. J. Perez, Bess Hardwick, Brendan Furneaux, Evgeny V. Zakharov, Dirk Steinke, Jeremy R. deWaard, Stephanie L. deWaard, Jaclyn McKeown, Norman Monkhouse, Cole Pace, Sujeevan Ratnasingham, Raina Schmuck, Amy Thompson, Jorge L. León-Cortés, Luciano Cagnolo, Julieta Soledad Arena, Jeremías Domínguez Masciale, Errol Douwes, Elli Groner, Masae Iwamoto Ishihara, Shunsuke Matsuoka, Mailyn A. Gonzalez, Alexandre Antonelli, Renata S. Sousa-Lima, Armin Niessner, Patrícia Guedes, Elizabeth G. Biro, Zuania Colón-Piñeiro, Dezene P. W. Huber, Santosh Mazumdar, Magda Bou Dagher Kharrat, Eddie P. Mondejar, Pablo Luis Tubaro, Eugene Marais, Marie-Eugénie Maggia, Darío Alejandro Lijtmaer, Sean W. J. Prosser, Andrea Murillo-Vázquez, Paola Pulido-Santacruz, Paula Leitman, Thiago Arnt Dorigo, Nao Ito, Paul Knoops, Maria L. Kuzmina, Hikaru Morishita, Keishi Kimoto, Dimby Raharinjanahary, Rhian J. Smith, Solny A. Adalsteinsson, Kevin G. Borja-Acosta, Paul Amouroux, Angela M. Mendoza-Henao, Angélica Diaz-Pulido, Anamaria Dal Molin, Luke L. Powell, Diego Inclan, Alice Calvente, David Dunson, Brian Fisher, Tomas Roslin, Otso Ovaskainen, Paul D. N. Hebert, Graham W. Taylor.
Resources: Ken A. Thompson, Sameer Padhye, Jayme Sones, Kate H. J. Perez, David Dunson, Brian Fisher, Tomas Roslin, Otso Ovaskainen, Graham W. Taylor.
Data curation: Emma Boehly, Ken A. Thompson, Jayme Sones, Kate H. J. Perez, Bess Hardwick, Brendan Furneaux, Dirk Steinke.
Writing – original draft: Emma Boehly, Scott C. Lowe, Ken A. Thompson, Graham W. Taylor.
Writing – review & editing: Emma Boehly, Scott C. Lowe, Ken A. Thompson, Anthony Fuller, Iuliia Zarubiieva, Brendan Furneaux, Dirk Steinke, Dezene P. W. Huber, Marie-Eugénie Maggia, David Dunson, Brian Fisher, Tomas Roslin, Otso Ovaskainen, Graham W. Taylor.
Visualization: Emma Boehly, Scott C. Lowe, Iuliia Zarubiieva, Graham W. Taylor.
Supervision: Scott C. Lowe, Paul D. N. Hebert, Graham W. Taylor.
Project administration: Ken A. Thompson, Iuliia Zarubiieva, Bess Hardwick, Rhian J. Smith, David Dunson, Brian Fisher, Tomas Roslin, Otso Ovaskainen, Graham W. Taylor.
Funding acquisition: David Dunson, Brian Fisher, Tomas Roslin, Otso Ovaskainen, Paul D. N. Hebert, Graham W. Taylor.
References
Data provenance
We identified a set of Malaise trap samples (bottles) that met two conditions: (i) the bottle was metabarcoded upon receipt via mild lysis and (ii) the bottle was entirely sorted for single-specimen barcoding, with any ‘excess’ (see below) recorded. Samples were derived from the LIFEPLAN project ((deWaard et al., 2024; Hardwick et al., 2024)) and the Global Malaise Program ([17]). Permissions to use each bottle were arranged with the data owners. Figure 3 summarizes the DNA barcoding data pipeline.
Core dataset generation and assembly
Raw metabarcode data were obtained as paired-end gzipped fastq files (i.e., .R1.fq.gz & .R2.fq.gz) accompanied by a plate map. Plate maps were converted into MAP parameter files and were analyzed with MAP v1.0 ([39]) with default settings for Illumina data. OTUs were queried against the June 2026 BOLDistilled COI library ([38]) to provide BIN matches and taxonomy.
Once tabular metabarcode results were obtained, we integrated the data with data from BOLD to produce a single table containing all relevant information from both sources. Data import, manipulation, and file handling throughout relied on the tidyverse ([54]) and readxl ([53]) packages, while retrieval of specimen-level Barcode Data Model (BCDM) records from the Barcode of Life Data System was performed with BOLDconnectR ([35]).
Processing proceeded independently for each sequencing lane. For every lane, filtered OTU tables and per-replicate read counts were loaded into memory. Next, BCDM records for all process IDs associated with the lane were retrieved. Nucleotide sequences from the BOLD SSB records were deduplicated and used to generate a FASTA file, which was then queried against BOLDistilled to obtain BIN assignments. This was done because the ‘true’ BIN assignments on BOLD cannot be efficiently replicated for metabarcode data due to (i) the RESL algorithm being proprietary, (ii) the use of private data on BOLD, and (iii) BOLD’s Barcode ID being only available via web browser and restricted to 100 sequences at a time during an exhaustive search. Since the reference library for metabarcode data becomes out of date every three months, metabarcoding OTUs assigned to Arthropoda were also matched to their corresponding BIN via the BOLDistilled lookup. Following BIN assignment, the metabarcode and single-specimen barcode data were joined by BIN in a single table that summarized total specimen counts and metabarcode read statistics within each bottle.
Derivation of the forward DNA amplification relationship
Equation (1) follows from the kinetics of a PCR run on a mixture of templates ([49, 37]), which Figure 4 traces for two BINs. Consider one PCR replicate of bottle , and let be the number of copies of BIN ’s amplicon after cycles, treating as continuous. We drop the indices and until the end of the derivation.
Figure 4. Amplification from different starting points at different rates. Two BINs amplified in the same PCR reaction (Eqs. (8) to (10)). BIN starts from more template (: heavier specimens, DNA more accessible, or both), but the primers bind BIN better (), so its copies grow faster and overtake those of BIN . When the shared reagents run out at cycle , both plateau at , up to the factor common to the reaction, and sequencing returns reads in proportion to these final copy numbers (Eq. (11)). Had the reaction stopped before the curves cross, BIN would have yielded more reads. The axes are schematic: a real reaction multiplies its template by many orders of magnitude.
Starting point. Each specimen releases DNA into the lysate, on average in proportion to its body mass (Elbrecht & Leese, [11]), and the reaction draws a fixed fraction of that lysate. BIN therefore enters the reaction with
copies of template. As is shared by every BIN of the reaction, the starting points of different BINs differ through the composition of the bottle () and through the body masses of their specimens ().
Exponential growth. In each cycle, a fraction of BIN ’s templates is copied. This efficiency depends on how well the primers bind the BIN’s sequence (Polz & Cavanaugh, [37]; Elbrecht & Leese, [11]), so it differs between BINs (Appendix § 13), but not from one cycle to the next while reagents are plentiful. After cycles,
so each BIN grows exponentially, at a rate between 0 (the primers do not bind) and (every template is copied in every cycle).
Plateau. The primers, nucleotides and polymerase are shared by every BIN in the reaction, and together they support at most some total number of copies. Idealizing saturation as an abrupt stop shared by every BIN, amplification ends at the cycle at which the total product reaches this budget,
and BIN ends with copies. The duration is thus set by the whole bottle: a bottle with more template, or with more readily amplified BINs, exhausts the budget in fewer cycles, so each of its specimens is amplified less. This is the bottle-level saturation of § 4.
Sequencing. The sequencer returns reads for the replicate, each drawn from the pooled product, so in expectation BIN receives reads in proportion to its share of that product,
The factor is common to every BIN of the replicate. Restoring the indices, Eq. (11) is Eq. (1), with the amplification term .
Consequences for the model. Two BINs of the same reaction compare as
so the read ratio confounds the ratio of specimen counts, which we want to recover, with the ratio of body masses and with a difference in amplification that grows with the duration of the reaction. A BIN that starts from less template but binds the primers better overtakes the other if the reaction runs long enough (Figure 4), and whether it does depends, through Eq. (10), on every other BIN in the bottle. The idealized amplification term factorizes into a per-BIN rate and a per-reaction duration. BIOTRAM keeps the product form of Eq. (1) but does not impose this factorization: its attention module predicts from the content of the whole bottle (§ 4.3), so the term can also absorb departures from the idealization ([49]).
Primer binding efficiency
Methods
Retrieving the binding sites. How well a BIN amplifies directly depends on its primer binding efficiency that itself depends on the sequences of its two primer binding sites. During metabarcoding PCR, the BF3 and BR2 primers must bind to the 20 nucleotides that flank the amplicon DNA sequence. Since those binding sites are replaced by the primer sequence itself, we cannot use the sequenced amplicons to determine their binding affinity. To accomplish this, we took two approaches.
The first route to obtaining primer binding sites used full mitogenomes, which contain both sites. We downloaded 11,080 complete mitogenomes from GenBank ([45]), restricting our search to Arthropoda records annotated as mitochondrial complete genomes. Records were queried and retrieved via the NCBI E-utilities using the rentrez R package ([55]), filtered to a minimum length of 14,503 bp (the smallest known mitogenome), and deduplicated to one record per named species. Sequences were parsed and written to per-species FASTA files using Biostrings ([36]). For each downloaded mitogenome, we obtained taxonomic information using the taxize R package ([5]), which queries the NCBI Taxonomy database. Species names were taken directly from each genome’s GenBank organism field, so only the ranked classification required an additional network query. For each mitogenome, we searched for binding sites of the degenerate COI primers BF3 and BR2 ([13]), allowing up to four mismatches per primer, using Biostrings. Sites overlapping ambiguous (N) bases were discarded, and where a primer matched more than one site in a genome, the best-scoring site was retained via pairwise alignment implemented in the R package pwalign ([1]). The amplicon between the retained BF3 and BR2 sites was extracted for each genome and written to a FASTA file, and candidate amplicons were filtered to a plausible COI length (388–424 bp) and in-frame codon length. 10,780 mitogenome records passed these filters and contain both sites, which gives forward and reverse sites for 9,396 BINs, of which 1,554 were in our dataset. When several genomes matched to one BIN, they were averaged.
The second route used full-length (658 bp) barcodes from single-specimen barcoding, which cover far more BINs, but only the forward site. The BF3/BR2 amplicon starts approximately 237 bases into the barcode region (median length 653 bases in BIOSCAN records), so the forward site lies inside a barcode record, whereas the reverse site lies beyond its 3’ end. Forward sites extracted from these records cover 91.5% of the BINs in our dataset. Forward sites were located in the full-length record using the same method as was used on mitogenomes.
From site to efficiency. Each site is scored into an efficiency between 0 and 1 by a function of its full 20-nucleotide sequence. For each retained amplicon, we estimated PCR efficiency based on the sequences of the binding sites and the primers, using the CalculateEfficiencyPCR function in the R package DECIPHER ([56]), which calculates the Gibbs free energy, formamide melt point, and hybridization efficiency of the primer and its binding site.
Imputation. BINs without a retrieved site receive predicted efficiencies. The forward and reverse efficiency, predicted for 7.4% and 98.8% of BINs, respectively, is an equal blend of a ridge regression on the one-hot encoded amplicon and a phylogenetic kernel ridge regression whose kernel is defined by the pairwise alignment identity between amplicons. Each BIN carries its two efficiencies, the logarithm of their product and, for each primer, a flag marking efficiencies taken from a mitogenome or from a barcode record of documented provenance.
A note for sequencing practice. Primer binding sites flank the amplified fragment and are discarded by short-read metabarcoding, yet the primer binding-efficiency block is one of the two largest contributions in our ablation (Appendix § 25). Extending the region sequenced by single-specimen barcoding by about 50 base pairs in each direction would make this information directly available in future datasets, and as read lengths grow, the limitation may resolve on its own. In general, for effective estimation of abundance, the marker used for single-specimen barcoding should remain longer than the one used for metabarcoding, ideally capturing both the forward and the reverse binding site.
Evaluation
Reference values. Over the 9,396 BINs with a mitogenome, the forward site has a mean efficiency of 0.64 and the reverse site 0.58, and 24% and 30% of BINs, respectively, fall below 0.4. Most sites match the primer exactly (91.5% of forward and 97.3% of reverse sites), but an exact match does not guarantee efficient binding. The forward and reverse efficiencies of a BIN are nearly independent (Pearson ), and efficiency differs between taxa. Among the BINs of our dataset with a retrieved forward site, the mean forward efficiency is 0.34 in spiders (Araneae) and 0.43 in Hymenoptera, with 72% and 52% of their BINs below 0.4, against 0.57 in Diptera and 0.68 in Lepidoptera. Over the 44,182 BINs with at least one specimen in the excess-free bottles, correlates with a BIN’s mean empirical at , and at over the 223 of them whose two sites both come from mitogenomes.
Coverage. Mitogenomes cover only 1,554 of the BINs in our dataset (1.2%), and they are the only source of reverse sites. Full-length barcode records supply a forward site for 91.5% of BINs, which carry 97.9% of the individuals in the excess-free bottles. In total, 92.6% of BINs, carrying 99.4% of those individuals, have a forward site read from a reference sequence. The forward efficiency is imputed for the remaining 7.4% (0.6% of the individuals), and the reverse efficiency for 98.8% of BINs.
Imputation accuracy. We train the barcode-based predictor on the 7,663 mitogenome BINs absent from our dataset and on 60,000 forward sites extracted from barcode records. We then evaluate it on the 1,508 mitogenome BINs present in it. The forward predictor explains 23% of the variance of forward efficiency (Pearson , Spearman , MAE 0.20), slightly less than a shrunken taxonomic mean (26%). The reverse predictor explains 16% of the variance of reverse efficiency (, , MAE 0.21), against 9% for the taxonomic mean, and keeps its advantage on families absent from training ( against 0.07). Its predictions are strongly shrunk toward the mean (standard deviation 0.09 for imputed reverse efficiencies, against 0.26 for those from mitogenomes), so an imputed value mainly carries the trend of the BIN’s relatives.
`Excess' specimens
During routine single-specimen barcoding (SSB) workflows, technicians regularly encounter samples with many individuals of what appears to be the same species (Figure 3). In such cases, it is neither practical nor cost-effective to barcode them all. These specimens are counted and then archived, and technicians note the microplate(s) on which some representative specimens were barcoded and remark on their taxonomy. The existence of excess will always be a feature of DNA barcoding data gathered in a resource-limited environment. This excess is recorded as a count and a taxon label, not as BINs, so the 603 of our 1,188 bottles that contain excess lack complete per-BIN ground truth, even though their metabarcoding data include the DNA of every specimen. We evaluate only on the 585 bottles without excess. Some of the others are used for training only; the allocation of the excess to specific BINs is described below.
Since the excess were metabarcoded, it is essential to accurately update the BIN specimen counts to reflect it. To accomplish this, we obtained a copy of the excess database from the CBG’s Specimens Unit, and filtered it to the bottles included in our dataset. To identify the BINs associated with excess specimens, we retrieved the records of all relevant microplates from the data previously downloaded with BOLDconnectR ([35]). This BOLD-derived dataset serves as the first reference point to identify candidate BINs. Finally, the cleaned excess dataset of bottle, taxon, and count fields were written to a standardized output file that served as the authoritative excess-count reference for all subsequent analyses.
After assembling a clean database of all BINs on relevant microplates and a clean excess database, we identified potential candidate BINs. After refining the manually-entered taxonomic information and outdated nomenclature, we queried the unique sequences against the BOLDistilled library using VSEARCH to assign BINs. BIN assignments were then rejoined to the data by sequence identity.
To generate a candidate list of BINs consistent with each excess record, the full taxonomic hierarchy (phylum through species) associated with every BIN recovered from a given sample was reshaped into long format and matched, case-insensitively and within sample, against the corresponding excess taxon label, allowing a label recorded at any taxonomic rank to be resolved to the appropriate BIN(s). Data manipulation, reshaping, and joins throughout were performed using the tidyverse collection of packages (Wickham et al., [54]). Excess records for which no candidate BIN could be identified within the focal sample set were flagged and proceeded through a secondary path (detailed below), while all successfully matched sample–taxon–BIN combinations were retained as the candidate excess dataset used in downstream specimen identification.
We then used the model itself to inform our excess assignment. We used a model trained only on no-excess bottles to predict the relative abundance of taxa in bottles with excess. (In the final testing set, only bottles with no excess were included.) For each BIN in the dataset, we quantified its “model prediction factor” (MPF) as its predicted abundance divided by its true abundance, quantifying the degree to which the model over- or under-predicted abundance for that BIN in that sample. Samples with a true abundance of zero but nonzero read support were assigned the minimum nonzero true abundance observed within the sample to avoid division by zero. Within each sample–taxon group, MPFs were rescaled to relative weights that summed to 1. Excess counts were assigned to each BIN in proportion to their relative weights to produce corrected ground-truth abundances.
For taxa that did not have a candidate BIN, we assigned them by matching the excess taxon against family, order, class, and phylum—in that order of precedence until at least one candidate could be found. All BINs sharing the matched higher rank were then treated as above and the same excess allocation procedure was applied. Results were then merged with the main dataset.
Body size estimation: pipeline and evaluation
Larger individuals contribute more DNA to the bulk DNA mixture, so the typical body size of a BIN is an important signal for our task. We estimate it in three steps, described below, and evaluate the estimates in Appendix § 15.§ 4.
Segmenting body area from specimen images
During the individual barcoding pipeline, specimens are photographed individually. Following image-based estimates of arthropod size and biomass (Schneider et al., [46]), we segmented 639,751 of these photographs. Our segmentation pipeline combines pretrained vision models without fine-tuning, as follows.
We first locate the specimen with an open-vocabulary detector (Grounding DINO; Liu et al., [30]). 2. The identified bounding-box is fed to two box-prompted segmentation models (SAM-HQ and SAM 3; Ke et al., [26]; Carion et al., 2026) which outline its overall shape (“envelope”; middle panels of Figure 5). We keep SAM-HQ’s mask, which traces finer detail, unless the two masks disagree (IoU below 0.8), in which case we keep the one with the higher predicted IoU; if SAM-HQ misses much of the specimen, we add the large regions that only SAM 3 finds. This way we prioritize the more detailed mask while avoiding the risk of missing parts of the specimen. 3. Next, we separate out the wings from the main body. To do this, we use two segmentation models focused on the body only: (a) we ask a vision-language model (Qwen3-VL-8B-Instruct; Bai et al., [2]) for a bounding box around the compact body (head, thorax and abdomen, ignoring spread wings), which SAM 2 (Ravi et al., [42]) then segments. (b) SegGPT (Wang et al., [52]) prompted with five hand-annotated examples. Spread wings add projected area but little tissue, so we remove them only when a wing-presence score is high. The score combines the vision-language model’s judgement of whether the wings are spread with the disagreement between the envelope and the body masks. We then cut from the envelope the regions that the body masks exclude: all of them if the two body masks agree (IoU at least 0.8); otherwise the larger of the two exclusions when the vision-language model or SAM 3 confirms the wings, and only the regions both exclude when neither does. The pixel count of the final mask, converted to with the image’s scale (known from the imaging system or read from a scale bar), is the specimen’s projected body area, a two-dimensional projection of its body. Averaging the logarithm of this area over the images of each BIN gives a per-BIN mean log area and a within-BIN spread for the 78,259 BINs with images, which cover 50.2% of the BINs in our dataset.
Figure 5 shows two examples of specimens with spread wings. The envelope (middle panels) is what a general-purpose segmenter returns: the organism’s outline, wings included. It is a faithful silhouette of the specimen as a whole, but it is a poor proxy for its mass. In particular, the wings provide a large cross-section of the image—for winged insects this is typically most of the surface area of the insect seen within the image—but contain very little tissue mass. We refine this segmentation mask with the wing carve described above, which keeps the envelope’s boundary only where the two body-only models agree that the pixel is not wing. How much this changes is not a detail: across the 631,161 scored specimens, 22% carry spread wings, and on those the carve removes a median 48% of the silhouette, reaching 71% at the ninetieth percentile. Over the whole set, including specimens with folded or absent wings, the median removal is 13%.
Estimating mass from segmented image surface area
Insect mass scales as a power of 2D body dimensions [44], so log mass should be linear in log area. We calibrate this relation on a separate collection of 26,114 dried and weighed specimens from 13,287 BINs. Weighed and imaged specimens are different individuals, so the relation cannot be fitted specimen by specimen; instead, we match the mean log mass of each genus to its mean log area, over the 332 genera with at least two specimens of each kind:
The offset is estimated hierarchically from class down to genus, each level shrunk toward its parent. The slope is shared across taxa, except in the orders with enough genera to estimate their own. Applying Eq. (13) to every imaged BIN gives a per-BIN dry mass.
Generalizing to unseen BINs
Half of the BINs in our data, and any BIN new to a monitoring program, have no image. To estimate their size as well, we fine-tune BarcodeBERT [33] to predict, from the BIN’s DNA barcode, both the BIN’s mean log area and the within-BIN standard deviation of that log area, with a squared-error loss on each. Genera and BINs held out for evaluation are excluded from training (Appendix § 15.§ 4). The same architecture trained on the allometric masses of step (2) predicts mass instead. Because the barcode carries nothing about an individual, its life stage or its sex, both networks predict the typical size of a taxon and the spread around it, not the size of any one specimen. The reported version of BIOTRAM reads the barcode-predicted log area and its spread for every BIN, including BINs with images; the measured areas are not among its inputs.
Evaluation
From area to mass. The fitted slope of Eq. (13) is 1.37 overall and ranges from 1.27 (Coleoptera) to 1.47 (Diptera) across the orders with their own slope. All are below 1.5, the value expected if mass scaled with the cube of a linear dimension and area with its square, that is, if body shape and density did not change with size. To test the relation on taxa it was not fitted on, we predict each genus with its family’s parameters withheld (five-fold cross-validation). Genus mean log mass is then predicted with Pearson () and a mean absolute error of 0.90 natural-log units, against 1.61 and 1.77 when each genus is given the mean mass of its order or the global mean. At the level of single BINs, the mean log area and mean log weighed mass correlate at across the 587 BINs that carry both measurements.
Body size from the barcode. On BINs held out from training whose genus is represented in training, the barcode network predicts the mean log area with Pearson (RMSE 0.84 natural-log units), against (RMSE 0.98) when each BIN is given the value of its nearest training barcode. On genera held out entirely it reaches (RMSE 1.12), against 0.70 (1.28) for the nearest barcode. The predicted within-BIN spread is only weakly related to the observed one (Pearson on held-out BINs), so it should be seen as a coarse taxon-level feature rather than a calibrated uncertainty.
Detailed model architecture
Expanding on the model overview in the main text (Figure 2), we show the full model architecture in Figure 6.
Batches. A training example is a whole bottle rather than a single (bottle, BIN) row, since both the amplification module and act on a bottle’s BINs jointly. A batch holds eight bottles, padded to the largest number of BINs among them and carrying a mask; padded entries take no part in the attention and are excluded from every term of Eq. (3).
Evaluation methods
BIOTRAM predicts a specimen count for each BIN in a bottle, and every metric derives from those same predicted counts. Some score the counts directly; others first normalize within the bottle to score the predicted composition. Since we care about the composition, about the counts themselves and about ecological usefulness, we report a set of metrics. Our primary metrics, JSD and MAE, are illustrated in Figure 7.
Figure 7. Calculation of mean absolute error (MAE), Bray–Curtis, and Jensen–Shannon divergence (JSD) between a true and predicted BIN composition. For a single bottle, the figure writes and for the true and predicted composition, and in Eq. (5), and for their mean. (A): Illustrative true (green) and predicted (blue) distributions over BINs. Their mean distribution is also shown (black line). Bray–Curtis is half the sum of BIN-wise absolute differences (red) between the two distributions, whereas MAE is the mean of the absolute differences. (B): To compute JSD, we take the entropy of each BIN composition. We compare the entropy of the mean distribution (grey bars) against the mean of the entropies for the true and predicted compositions (black line). JSD is their difference (red), summed over BINs. Note that larger relative errors (BINs 4 and 6) contribute much more to the total JSD than smaller relative errors (BINs 2 and 1) despite having similar sized absolute difference between the predicted and true composition.
Jensen–Shannon divergence (JSD), the primary metric, defined with the composition term of the objective (§ 4.4). It turns predicted counts into proportions and measures how far the predicted composition sits from the true one. Figure 8 illustrates how this is calculated.
Mean absolute error (MAE) per bottle. Taken within a bottle and averaged across bottles. This is the target quantity in its own units. Table 1 and Table 3 also split it by truth, over all test rows: on rows with no individuals, on the rest, because a model can win the aggregate purely by predicting zeros well.
MAE on the log scale (MAE-log), the same error on log counts. Counts span four orders of magnitude, so MAE is dominated by the abundant rows and MAE-log scores proportional rather than absolute error.
Presence AUC, whether truly present BINs rank above absent ones; 0.5 is chance. The score ranked is the predicted count , for BIOTRAM exactly as for every baseline.
Spearman correlation per bottle, whether a bottle’s BINs are put in the right order; Table 1 and Table 3 compute it over all test rows at once.
Shannon diversity , predicted against observed within-bottle Shannon diversity from single-specimen barcode data; a standard -diversity quantity, and a direct test of whether the predicted composition is the right shape rather than merely close row by row.
Bray–Curtis Mantel correlation, between-bottle dissimilarity on family-level abundances; a standard quantity in -diversity analysis, assessing whether two bottles are correctly judged similar. It is computed at family level, because at the global scale of the dataset two bottles often share no BIN.

Figure 8. Jensen–Shannon divergence calculation on the 2-simplex. (A): The 2-simplex represents all possible distributions (summing to one) across three possible bins. The corners of the simplex correspond to distributions comprised of only a single bin, edges mixtures between two bins. The entropy of each possible distribution is shown (heatmap), with the centre point [1/3, 1/3, 1/3] having the highest entropy and vertices [1, 0, 0], etc., the lowest. We indicate the distributions for three sample distributions, , , and . (B): The Jensen–Shannon divergence between distributions and is the difference between the entropy at their midpoint, , and the mean of their individual entropies. (C): As with A, except the heatmap shows contours of equal JSD from distribution . Note that distribution is closer to than is under the JSD measurement, despite being further away in a Euclidean sense. This difference arises because underestimates bin 2 by too much; the true value of bin 2 in is very surprising given the estimate in .
The last two are the quantities an ecological study would actually report, and they better represent the model’s usefulness in practice.
Full baseline comparison
Table 1 in the main text trains every deep model on the objective of Eq. (3), each with the auxiliary weights tuned for it (Appendix § 24). Table 3 gives every deep model the same weights, those selected during development
(Appendix § 24), so that only the architecture differs, and Table 4 lets every model use the objective it does best under, which answers the more practical question of what the best available predictor is. The two are not comparable to one another and should not be read as a single ranking. Figure 9 shows both in bar form.

Figure 9. The comparison of Table 3 and Table 4 in bar form. Bars start at zero, and error bars show one standard deviation across seeds. Rows are grouped because the groups are not comparable to one another: only the first holds the training objective fixed with no external model on either side, and so only it speaks to the architecture. A starred bracket spans rows that are not distinguishable by a two-sided Welch test over seeds at ; it is drawn for the matched-objective group only, since elsewhere the objective differs between rows and such a statement would not be like-for-like. On the counts BIOTRAM is clear of four of the four matched baselines and on composition of four; the read-blind variant shown beside it is clear of two on the counts and of three on composition. BIOTRAM is level with TabICLv2 on count accuracy, 0.6% apart, at 35% of its parameters.
Architecture only. With the exact same loss, BIOTRAM’s MAE per bottle is 5.8% below that of the strongest baseline, against 3.2% when each model has its own weights, and its JSD differs from the best baseline’s by , against . Split by truth, over all rows, the MAE gain is 13.5% on rows with no individuals and 7.5% on the rest, against 5.4% and 3.9% with tuned weights. Of the four baselines, BIOTRAM is ahead of four on MAE and four on JSD; on rank correlation it is ahead of two and level with two, and on presence AUC it is level with three of them (Table 4).
The hurdle mixed model of Furneaux et al. [14]. They relate metabarcoding reads to individual counts in 45 Malaise-trap bottles with two generalized linear mixed models: a logistic regression of whether a BIN detected by metabarcoding is present as an individual, and a zero-truncated Poisson regression of how many individuals it has when it is. Both take the logarithms of the BIN’s reads, pooled over PCR replicates, and of the bottle’s total read count, with random intercepts nested by order, family and genus and no effect at a rank the BIN is not identified to. Their unit of analysis, a BIN with reads in a bottle, is exactly one of our rows, so we fit both models to our training rows with the software they used (glmmTMB 1.1.13) and predict each row by the hurdle’s mean, the presence probability times the truncated mean. The specification is fixed, so nothing is tuned, and the model sees only its published predictors. Under the three shifts of Table 2 it scores 0.2141 (spatial), 0.2366 (temporal) and 0.2267 (lane), behind the best matched baseline in each.
TabICLv2 and foundation-model pretraining. TabICLv2 (Qu et al. [40]) is kept apart from the other baselines in Table 1, Table 3 and Table 4, partly because of its size: its 28.5M parameters are almost three times as many as BIOTRAM’s, but also because TabICLv2 is a transformer pretrained on synthetic tabular datasets and used in context: without any gradient step on our data, it predicts each test row by attending to the training rows. It therefore brings prior knowledge from pretraining and direct access to the training set at inference, which no other model has. The pretraining matters: the same architecture trained from scratch on our data reached a JSD of 0.180, against 0.128 for the pretrained model in the same setting. TabICLv2 has the best presence AUC (0.924), MAE-log (0.226) and MAE (0.249) of any model in our tables. BIOTRAM is level with it on count accuracy (MAE 0.699 against 0.704, 0.7% apart) and ahead of it on composition, with 35% of its parameters, while remaining behind it on MAE and rank correlation and on presence AUC (Table 1). Attending to the training rows also has a price at inference: a bottle costs TabICLv2 floating-point operations against BIOTRAM’s , a factor of .
Error by abundance and by novelty
Table 5 and Table 6 break the error of Table 3 down by test row: by true specimen count, and by whether the BIN occurs in any training bottle. Both compare BIOTRAM with the four matched baselines only. Table 5 scores MAE on , so that errors on very different counts can be compared; Table 6 scores it on the counts.
| True count | Rows | % | BIOTRAM ↓ | Transformer ↓ | CNN ↓ | MLP ↓ | DeepSets ↓ |
| 0 (absent) | 14,449 | 53.6% | 0.173 | 0.194 | 0.199 | 0.200 | 0.214 |
| 1 | 8,373 | 31.1% | 0.214 | 0.211 | 0.214 | 0.216 | 0.263 |
| 2–5 | 3,275 | 12.2% | 0.428 | 0.416 | 0.416 | 0.416 | 0.388 |
| 6–20 | 677 | 2.5% | 0.681 | 0.663 | 0.694 | 0.710 | 0.627 |
| 21–100 | 146 | 0.5% | 0.609 | 0.631 | 0.701 | 0.672 | 0.568 |
| 100+ | 14 | 0.1% | 0.415 | 0.446 | 0.542 | 0.635 | 0.498 |
Table 5. Error by true abundance, at the matched objective: MAE on for BIOTRAM and for all four baselines of Table 3, each trained on BIOTRAM’s own loss with the same inputs; columns ordered by overall MAE. Bold is the best value in each row and underline every value a two-sided Welch test over seeds cannot separate from it at , or the second-best value where it separates them all.
| Test rows | Rows | BIOTRAM ↓ | Transformer ↓ | CNN ↓ | MLP ↓ | DeepSets ↓ |
| Seen in training | 21,903 | 0.853 | <u>0.941</u> | 0.955 | 1.004 | 1.068 |
| Never seen | 5,031 | 0.373 | 0.376 | <u>0.357</u> | 0.343 | 0.467 |
Table 6. Test rows split by whether the BIN appeared in training, at the matched objective. MAE in occurrences, for BIOTRAM and the baselines of Table 3; marked as in Table 5.
By abundance. BIOTRAM’s structure acts most directly at the two ends of the range. Rows with no individuals, the most frequent bucket, are the ones the presence gate of Eq. (2) can remove outright. On counts above 100, BIOTRAM’s error is 7% below that of the Transformer, the matched baseline with the lowest overall MAE, a difference within seed noise. Few rows fall in this bucket, but they hold a large share of the specimens, so they weigh most on any estimate of total abundance or biomass. Between 1 and 100 specimens, BIOTRAM trails at least one matched baseline on three of the four buckets.
By novelty. Because the splits are by bottle, 19% of test rows carry a BIN that occurs in no training bottle. Their per-BIN quantities cannot be learned from training rows and must be inferred from the barcode and the taxonomy. Most of these BINs are rare, so every model makes a smaller error on them than on seen BINs. On seen BINs, BIOTRAM’s MAE is 0.853 against 0.941 for the Transformer. On unseen BINs, where errors in the barcode-based estimates pass directly into the count, it is behind two of the matched baselines, level with one and ahead of one.
What a prediction looks like
Figure 10 shows the predictions themselves for one seed of the reported arm. The left panel groups every test row by prediction and truth. The quantity is each BIN’s share of the bottle it is in. The right panel aggregates each bottle to its Shannon diversity and compares BIOTRAM against a linear regression that sees only read counts, the BINs and the bottle’s total; each carries its own least-squares fit. The slope is the part worth reading: an estimator can have a respectable while systematically flattening diversity toward the mean, so we are looking for the model closest to the 1:1 line.
Figure 10. Predictions, row by row and bottle by bottle. One seed (200) of the reported arm on the test split: 26,934 rows across 89 bottles. Left: relative abundance per BIN and bottle, predicted against true, as a two-dimensional histogram on a axis in percent; colour is the number of rows in a cell, on a logarithmic scale, and the dashed line is 1:1. The cell at the origin holds every row that is zero in both, more than half the dataset; the colour scale is pinned to the largest cell beside it, so that one saturates (the arrow on the bar) rather than pushing the rest into the dark end. Right: Shannon diversity per bottle for BIOTRAM (blue, slope 0.95, 0.89) and for an ordinary least-squares regression of the specimen count on log read count and the log of the bottle’s total read count (orange, slope 0.86, 0.70), each with its own fitted line against the 1:1 diagonal.
Calibration of the predictive distribution. The negative-binomial head gives a full predictive distribution for each count, not only a point estimate, so we check and score the distribution itself on the held-out rows over 12 seeds. For a discrete target, the plain probability integral transform is biased upward when about half of the rows are exactly zero (its mean here is 0.804), so we use its randomised mid-point version, whose mean is 0.500 under calibration. It averages 0.520: the predictive distribution is centred close to the observations. It is too wide, however, as nominal 90% intervals contain 0.967 of the observations. Part of this excess is unavoidable: on counts that are mostly zero or single-digit, a discrete interval cannot cover exactly 90%.
Distribution shift
Regimes. Each shift regime of Table 2 replaces the random split by one that holds out a whole stratum of bottles for testing: the bottles of a region never sampled in training (spatial), those collected after every training bottle (temporal), or those of a sequencing batch absent from training (lane). Every deep model is retrained on the remaining bottles with the matched objective, so the rows of Table 2 read against Table 3. The bottles with excess that fall inside the held-out stratum are dropped as well, which leaves 694 (spatial), 702 (temporal) and 684 (lane) training bottles instead of 864.
Reading the table. Each row compares BIOTRAM with the best baseline of that row alone, chosen among the four matched deep baselines and the hurdle mixed model, so different rows can name different models: the best baseline in distribution need not be the best one under a given shift. The baseline of a row is chosen on JSD, and its MAE per bottle is shown beside BIOTRAM’s. is each model’s change in JSD from its own random-split score (Table 3 and Table 4), so the of a row’s baseline is not measured from the Random row whenever the two rows name different models.
Results. The shifts cost every model far more than the differences between models: BIOTRAM loses +0.0419, +0.0662 and +0.0517 JSD on the spatial, temporal and lane regimes, while its margin over the best baseline of each regime is −0.0090, −0.0099 and −0.0047, each tested with the same two-sided Welch test over seeds () as in § 5. The hurdle mixed model of Furneaux et al. [14] scores 0.2141, 0.2366 and 0.2267 JSD under these three regimes, behind the best matched baseline in each. On counts, BIOTRAM has the lower MAE per bottle in every row of Table 2, including the collaterals dataset, where the hurdle mixed model is closer on composition.
Validation on external “collaterals” dataset
Our main results are evaluations using splits of data collected from one metabarcoding pipeline in one laboratory only. We also scored model performance on a “collaterals” dataset, comprised of fifteen Malaise-trap bottles from another country (Sweden), processed by a different laboratory. The dataset is unpublished and was shared by
collaborators. Each bottle was metabarcoded in a single PCR replicate, without location, date or wet-biomass records, from two DNA extractions of the same material scored separately: a mild lysis (L), as in our data, and a homogenate (H). The specimens were also barcoded individually, so each bottle’s true composition is known. Table 7 compares BIOTRAM, one of its variants, the four matched deep baselines, TabICLv2 and the hurdle mixed model of Furneaux et al. [14] on these bottles; with fifteen of them, only large differences are interpretable.
| Model | JSD ↓ Lys. | JSD ↓ Hom. | MAE/bottle ↓ Lys. | MAE/bottle ↓ Hom. | Presence AUC ↑ Lys. | Presence AUC ↑ Hom. |
| Read share (sanity check) | 0.277 | 0.237 | 3.38 | 2.86 | 0.772 | 0.839 |
| Uniform (sanity check) | 0.255 | 0.255 | 3.12 | 2.87 | 0.540 | 0.504 |
| BIOTRAM | 2.78 | 2.54 | 0.722 | 0.740 | ||
| BIOTRAM trained one replicate at a time | 2.98 | 2.48 | 0.745 | 0.760 | ||
| Transformer over the bottle | 2.68 | 2.37 | 0.730 | 0.723 | ||
| DeepSets | 3.04 | 2.62 | 0.704 | 0.695 | ||
| 1-D CNN on the barcode | 2.53 | 2.34 | 0.681 | 0.658 | ||
| MLP | 2.59 | 2.38 | 0.692 | 0.650 | ||
| TabICLv2 (in-context) [40] | 2.40 | 2.19 | 0.735 | 0.739 | ||
| Hurdle GLMM [14] | 0.203 | 0.197 | 4.15 | 3.98 | <u>0.770</u> | <u>0.823</u> |
Table 7. Table 7: Extended results on external Collaterals dataset. 15 Malaise-trap bottles from a different country, processed by a different laboratory, with one PCR replicate per bottle and no bottle-level covariates (Appendix § 22). Lysis (Lys.) and homogenate (Hom.) are two DNA extractions of the same bottles, scored separately. JSD is the mean over seeds one standard deviation across seeds; MAE/bottle (as in Table 2) and presence AUC (over all rows) are means over seeds. Every model sees identical rows. Extends bottom half of Table 2. Bold: best model; underlined: second-best.
Scoring. 534 of the BINs are absent from the 74,365-BIN vocabulary the models were trained on: they receive BarcodeMamba embeddings, taxonomy mapped through the training vocabulary, and body-size and primer-efficiency predictions from the same models as every other BIN (§ 4.1).
Results. Assigning each bottle’s specimens in proportion to its reads gives a JSD of 0.277 on L and 0.237 on H, and spreading them uniformly over the detected BINs gives 0.255. BIOTRAM reaches 0.219 and 0.218 over twelve seeds, and its seed-averaged prediction is closer to the truth than the read shares in thirteen and twelve of the fifteen bottles. By a two-sided Welch test over seeds, it is level with the Transformer and TabICLv2 on both extractions and ahead of DeepSets, the 1-D CNN and the MLP, which do worse than the read shares. It is behind the hurdle GLMM on composition (0.203 and 0.197) but clearly ahead on count error (MAE/bottle 4.15 and 3.98 against BIOTRAM’s 2.78 and 2.54). With fifteen bottles that ordering is not firm: a bootstrap over bottles puts the mixed model’s margin over BIOTRAM at and . A variant trained with each PCR replicate as a separate sample, the format of this dataset, is level with BIOTRAM (0.224 and 0.217).
The latent quantities
Do the predicted quantities mean what they are named? The model’s distinguishing property is its intermediate quantities that are supposed to correspond to physical quantities and give meaningful explanations of the model’s prediction. Since the primer efficiency and body mass receive the corresponding external estimates as input features, we cannot correlate them against those estimates without circularity. Instead, we can check that the empirical per-BIN log(reads/specimen), measured from the metabarcoding pipeline, correlates with the predicted (Eq. (1)), averaged per BIN over the test bottles. The module also sees the BIN’s own read, so part of that association is mechanical rather than discovered, and the read-blind variant is the cleaner test of the same question.
Over 12 seeds of BIOTRAM at the development weights (Table 3), it reaches Pearson +0.627 0.007, against +0.593 0.013 for alone and +0.444 0.025 for alone. The read-blind variant, whose amplification term never sees the read, still reaches +0.461 0.013 (+0.397 and +0.411 for the two terms alone, 12 seeds). Figure 11 shows the distribution of both quantities across the test set and their agreement with the external measurements at the level of taxonomic order. Note that the objective supervises toward this quantity at weight 0.05, so the number is a generalization measure over held-out bottles rather than an unsupervised discovery.

Figure 11. The latent quantities of the reported model. Left: log DNA amplification; right: body mass. Top: their distributions across the test set; axes are clipped to the central 99% and the count outside is stated. Bottom: the same latent quantities (blue) against external measurements (orange), aggregated to taxonomic order and expressed in standard deviations across orders so that both sit on one axis.
Loss-weight tuning
Principle. A baseline comparison is only fair if every model’s objective is tuned with the same care, since an under-tuned baseline makes the proposed model look better than it is [57]. The three auxiliary weights (, , ) of Eq. (3) are therefore tuned for each of the five models separately, under one protocol fixed before any run and with the same budget. The likelihood is the negative binomial of Eq. (4) for every model.
Search. Every model runs the same four stages. (i) A full factorial over the three weights, on the logarithmic grid , and . (ii) At the best of these, each weight set to zero in turn, and KL in place of JSD for the composition term. (iii) A second factorial at half the step, around the best configuration so far; a weight set to zero in (ii) is tested at zero against two values below the smallest one of (i), so that zero is found to be optimal rather than assumed. (iv) The learning rate at a third of and at three times the model’s tuned value. Stage (iii) is a factorial rather than one weight at a time because the weights interact: four of the five models have a two-way interaction that is significant in a factorial ANOVA of stage (i), which a search moving one weight at a time can walk past. A best configuration on the edge of the range explored for a weight receives up to two further steps outward, under the same rule for every model.
Budget. Every model is given the same 61 configurations, plus between zero and four boundary steps granted by the rule described above: 61 to 65 configurations per model. BIOTRAM’s 62 configurations lie inside the range of the baselines (Table 8). Each configuration is trained on the same five seeds, 1,570 runs in all. The budget is counted in configurations and seeds.
| Model | Budget configs | JSD | MAE/bottle | AUC | |||
| Transformer | 63 | 1 | 0 | 0.1 | 0.1257 0.0019 | 0.723 0.020 | 0.9156 0.0017 |
| 1-D CNN | 61 | 2 | 0.009375 | 0.2 | 0.1235 0.0008 | 0.762 0.016 | 0.9145 0.0008 |
| MLP | 63 | 1 | 0 | 0.00625 | 0.1234 0.0006 | 0.737 0.025 | 0.9154 0.0007 |
| DeepSets | 65 | 1 | 0.0375 | 0.00625 | 0.1256 0.0010 | 0.752 0.036 | 0.9162 0.0013 |
| BIOTRAM | 62 | 1 | 0.15 | 0 | 0.1200 0.0009 | 0.699 0.017 | 0.9213 0.0006 |
Table 8. Every model at its own loss weights. The three auxiliary weights of Eq. (3) are tuned separately for each model, with the same protocol and budget (Appendix § 24): every model is given the same 61 configurations, plus up to two additional steps for any weight whose best value lies at the edge of the explored range, each trained on the same five seeds. Test metrics at the selected weights, mean standard deviation over eight seeds disjoint from the search; the test set played no part in selection. We highlight the best and _second best_ model.
Selection. A configuration is scored on validation JSD, MAE and presence AUC together. Each metric is divided by its own seed-to-seed standard deviation, pooled over the stage-(i) factorial, so that the three enter on a common scale, and they are averaged per seed and then over seeds into a composite ; the configuration with the lowest is selected. A configuration is admissible only if none of the three metrics is worse than at the centre of the stage-(i) grid by more than one noise band, so the composite cannot trade one metric for another.
Confirmation. The selected weights of each model are retrained on eight seeds disjoint from every seed used in the search, and from those on which the model’s architecture was selected. The test set plays no part in any selection; Table 8 and Table 1 in the main text report it on these retrained models.
Result. With every model at its own weights, BIOTRAM is ahead of the best baseline on the three test metrics: JSD 0.1200 against 0.1234 for the MLP (), MAE per bottle 0.699 against 0.723 for the Transformer (), and presence AUC 0.9213 against 0.9162 for the DeepSets model (; two-sided Welch tests over seeds). The selected weights differ between architectures (Table 8): the MLP and the Transformer do best with no presence term, zero beating both smaller values, whereas the 1-D CNN and DeepSets keep a small one, and BIOTRAM keeps its presence term and does best without . No single weighting suits every model, which is why each is tuned separately. Everywhere else in the paper, except in Table 1, BIOTRAM and the matched baselines use the weights selected during development, , rather than those tuned here. BIOTRAM’s tuned weights reach 0.1200 test JSD, slightly behind the 0.1194 its development weights reach in Table 3. That is expected rather than contradictory: a configuration is selected on a composite of validation JSD, MAE and presence AUC, never on test JSD, so the returned configuration need not be the one with the lowest test JSD — and the comparison in this appendix holds every model to that same rule.
Ablation study
We ran a hierarchical ablation over 20 capability blocks. Each block is removed from the reported configuration on its own, and the model retrained, paired seed-for-seed against a control trained on the identical split; the screen runs at four seeds and survivors are confirmed at eight. Table 9 reports the four-seed screen on the test set, grouped by the kind of component: for each metric, the mean paired difference from the control and the significance of a paired -test.
| Component removed | JSD | MAE/bottle | presence AUC |
| Architecture | |||
| Barcode CNN and embedding adapter | +0.0085*^{*} | +0.040^{*}** | −0.0031^{*} |
| Cross-BIN attention (each BIN processed alone) | +0.0054^{} | +0.046^{*} | −0.0024^{} |
| Taxonomy embedding | +0.0034^{*}** | +0.012^{*}** | −0.0016^{*}** |
| Read-informed presence heads | +0.0019 | +0.023^{*}** | −0.0029^{*} |
| Frozen barcode embedding | +0.0014 | +0.004 | −0.0011 |
| Inputs | |||
| Primer binding efficiencies | +0.0127^{*} | +0.026^{} | −0.0242^{*} |
| Extra training bottles with excess | +0.0117^{*} | +0.020^{} | −0.0043^{} |
| Bottle’s total specimen count | +0.0037^{} | +0.073^{*} | −0.0052^{} |
| BIN’s own read count in the amplification module (read-blind variant) | +0.0039^{} | +0.048^{} | −0.0019^{*}** |
| Bottle and primer inputs to the presence head | +0.0018^{} | +0.004 | −0.0012^{*}** |
| Barcode-predicted body size | +0.0015^{*}** | +0.008 | −0.0006 |
| Per-replicate sequencing depth | +0.0008^{*}** | +0.014 | −0.0011^{*}** |
| Pretrained geographic encoder (sinusoidal encoding instead) | +0.0012^{*}** | −0.006 | 0.0000 |
| Loss | |||
| Composition loss | +0.0023^{} | +0.022^{*}** | −0.0015^{*}** |
| Count objective (compositional head instead) | +0.0027^{*}** | N/A | N/A |
| Presence loss | +0.0013^{*}** | −0.003 | −0.0018^{*}** |
| Reads-per-specimen loss | +0.0007^{*}** | +0.005 | −0.0003 |
| Per-BIN dispersion (one global value instead) | +0.0007 | +0.007 | −0.0001 |
| Regularization | |||
| Exponential moving average of the weights | +0.0052^{} | +0.051 | −0.0032^{} |
| Online augmentation | +0.0014^{} | +0.027^{*}** | +0.0010^{*}** |
Table 9. Table 9: Model component ablations. Each row removes one component of BIOTRAM, retrains the model and compares it with the full model on the test set, paired seed for seed over four seeds (full model: JSD 0.1198, MAE/bottle 0.700, presence AUC 0.9136). is the mean paired difference, ablated minus full, so a positive JSD or MAE/bottle, or a negative AUC, means that the component helps. Values in bold differ from zero at in a two-sided paired -test (*: , **: , ***: ). Rows are ranked by the -value of JSD within each group. N/A: a compositional head predicts no counts to score.
What carries the model. Removing 17 of the 20 blocks in Table 9 raises test JSD significantly (, uncorrected). The two largest effects are data and chemistry rather than architecture: the primer binding efficiencies (+0.0127 test JSD) and the extra training bottles with excess (+0.0117). The learned components on top of the frozen encoder follow (+0.0085), then the cross-BIN attention of the amplification module (+0.0054), and giving it the BIN’s own reads is worth +0.0039 on its own.
Presence supervision. Removing either form of presence supervision, the presence loss or the read-informed presence heads, moves test JSD and the presence AUC of Table 9 by less than 0.003, but costs a large part of the AUC of the presence probability (−0.084 and −0.053).
Identity is redundant. Removing the frozen barcode embedding while leaving the taxonomy embedding in place costs +0.0014 test JSD (), and removing the taxonomy embedding while keeping the barcode costs +0.0034 (). The first result is consistent with the taxonomy being able to stand in for the barcode: genus is unknown for 76.9% of BINs and species for 87.5%, so for three quarters of the BINs a taxonomy lookup resolves only to family or coarser, yet what it does resolve is apparently enough to cover most of what the barcode carries.
Model scaling against baselines
Capacity. Model capacity is not what separates the models. Every architecture is retrained at each rung of a capacity ladder, 200k to 128M parameters for the baselines and 3.2M upward for BIOTRAM, and tuned again at every rung (Figure 12). Capacity moves each of them by at most 0.0062 JSD across its own ladder, while BIOTRAM’s margin over the best baseline of the same rung stays between 0.0018 and 0.0052: BIOTRAM at its worst size (0.121) is still ahead of the 1-D CNN at its best (0.123).

Figure 12. Accuracy against cost. Test JSD of each architecture against its training/inference demands: (A) number of trainable parameters; (B) the compute requirements of one training run; (C) and the compute requirements of one prediction (note the discontinuity in the x-axis here). FLOPs are measured on the real forward and backward passes. We explore baseline models up to 128M parameters to ensure we explore a compute budget at least as high as that used for BIOTRAM, but find benefits from increased size are saturated before reaching this. TabICLv2 does not require any additional training and is deployed with in context learning only; consequently its inference cost is much larger—more than 100,000 times that of the largest BIOTRAM model we explore.
Compute. Figure 12 also places the same models against the cost of training one model and of predicting one bottle, both counted in floating-point operations rather than in wall clock, so that they do not depend on the hardware. At the 10.5M rung BIOTRAM trains for 4.4 the compute of the deep baselines. Predicting one bottle costs it 1.4 operations, inside the range the deep baselines span at that rung (6.1 to 9), so cost per prediction does not follow the parameter count. TabICLv2 has no training cost on our data and instead attends to every training row for each prediction, at 5.4 operations per bottle.
Training-set size. Every model is also retrained on nested subsets of the 864 training bottles: the first bottles of one fixed random permutation, for six values of from 54 to 864, so that each subset contains the smaller ones and all models at a given see the same bottles. Validation and test bottles are unchanged, and everything fitted on the training data, such as the feature standardisation or TabICLv2’s context, is refitted on the subset. Each model keeps a single size and the configuration tuned on the full training set (Figure 13). Every curve is still falling at the full training set, and BIOTRAM’s most steeply, from a JSD of 0.1841 to 0.1198: it is behind every other model up to 216 bottles, and from 648 bottles on it is ahead of the deep baselines, level there with TabICLv2 and ahead of it on the full training set as well (0.1198 against 0.1227, ). Its structure therefore does not make BIOTRAM more data-efficient at this size; it pays off once there are enough bottles to learn the per-BIN quantities it relies on.
Figure 13. Accuracy against training data volume. Test JSD of every model retrained on nested subsets of the training bottles (log scale), each with the configuration tuned on the full training set and at a single size: BIOTRAM’s own for the four deep baselines, as in Table 4, and 28.5M parameters for TabICLv2. Error bars show one standard deviation across three seeds. With all 864 bottles, BIOTRAM reaches 0.1198 against TabICLv2’s 0.1227 ().
How a read reaches the count
Read elasticity. We measure how a change in a BIN’s reads propagates to its predicted count, as the elasticity . In the read-blind variant, Eq. (2) passes a change in reads to each replicate estimate one for one (measured: 1.00), and the learned replicate merge takes back 0.60, for an end-to-end elasticity of 0.40 (3 seeds). The merge sees only the three replicate estimates and the bottle’s specimen total, so this discount cannot depend on the bottle’s total read count or on the other BINs. In the reported model, the amplification term also responds to the BIN’s own read: its derivative with respect to the BIN’s own read inputs (log read count and read share) has a per-seed mean absolute value between 0.37 and 0.43 over 12 seeds, at least 190 times its derivative with respect to another BIN’s read (0.002). On average, when a BIN’s reads double, its predicted amplification rises with an elasticity of 0.29, so part of the extra reads is attributed to stronger amplification rather than to more individuals. The replicate estimate then grows with an exponent of 0.71 instead of one, the merge takes back 0.40, and the end-to-end elasticity is 0.31, below the read-blind variant’s 0.40. With the read in view, the discount is larger and can vary between BINs and with the bottle’s total read count. Every baseline can fit such an interaction, since each row it receives carries the log read count, the BIN’s share and the bottle’s total read count; the read-blind variant gives it up by construction.
What the per-BIN read contributes … Estimating the amplification without the BIN’s own read keeps the two sides of Eq. (1) strictly separate, at a cost of +0.0036 JSD (Table 3). The read-blind variant is still ahead of two of the matched baselines and level with two on MAE per bottle, and ahead of three and level with one on JSD.
Attention across BINs. The amplification module is the only component in which BINs attend to one another; it was designed to express competition between BINs for shared reagents and sequencing budget. To test whether it uses this attention or mainly the BIN’s own read, we make five comparisons, each paired over six seeds. Three compare the module with one of identical width, depth and feed-forward structure in which BINs do not attend to one another, varying which reads both see; every variant receives the total read count of the three replicates. The other two remove that total: one repeats the first comparison without it, the other measures what it adds to the module without attention. When the BIN’s own read and those of the other BINs are visible, attention lowers JSD by 0.0033 (6/6 seeds, ; each in this paragraph is a paired -test, Holm-corrected across the five paired comparisons). Masking only the BIN’s own read removes this gain (−0.0004 JSD, 3/6 seeds, ), and with no reads at all, attention has no measurable effect (3/6 seeds, ). Without the replicate’s total read count, attention over BINs with reads changes JSD by −0.0020 (6/6 seeds, ), and the total read count without attention changes it by −0.0001 (3/6 seeds, $p = 1). Attention therefore pays off only when the module sees the BIN’s own read, and most when it also sees the replicate’s total read count, which does nothing without attention. This suggests that it exploits an interaction between the per-BIN reads and the total reads rather than competition between taxa.
References
- [1]Aboyoun, P., & Gentleman, R. (2024). pwalign: Perform pairwise sequence alignments. <a href="https://doi.org/https://bioconductor.org/packages/pwalign" rel="noopener noreferrer" target="_blank">https://doi.org/https://bioconductor.org/packages/pwalign</a>
- [2]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hu, B., Jiang, S., Li, Z., Li, M., … Zhu, K. (2025). Qwen3-VL technical report. https://doi.org/10.48550/arXiv.2511.21631
- [5]Scott A. Chamberlain and Eduard Szöcs. taxize: taxonomic search and retrieval in R. F1000Research, 2:191, 2013. doi: 10.12688/f1000research.2-191.v2. 19DOI
- [6]K. Chen, Z. Hu, Z. Xia, D. Zhao, W. Li, and J. K. Tyler. The overlooked fact: fundamental need for spike-in control for virtually all genome-wide analyses. Molecular and Cellular Biology, 36(5):662–667, 2016. doi: 10.1128/MCB.00970-14. 9DOI
- [7]Robert K. Colwell, Laura Melissa Guzman, Dirk Steinke, Anne Chao, Daniel H. Janzen, Winnie Hallwachs, Austin Baker, José L. Fernández-Triana, Paul D. N. Hebert, Frank Joyce, Robert Puschendorf, Donald L. J. Quicke, Rodolphe Rougerie, M. Alex Smith, Nelson Zamora, and Michael J. Sharkey. Constructing a lower-bound estimate of the global number of insect species on a hyperdiverse empirical foundation. Proceedings of the National Academy of Sciences, 123(27):e2524283123, July 2026. doi: 10.1073/pnas.2524283123. 2DOI
- [10]Didham, R. K., Basset, Y., Collins, C. M., Leather, S. R., Littlewood, N. A., Menz, M. H. M., Müller, J., Packer, L., Saunders, M. E., Schönrogge, K., Stewart, A. J. A., Yanoviak, S. P., & Hassall, C. (2020). Interpreting insect declines: seven challenges and a way forward. Insect Conservation and Diversity, 13(2), 103–114. https://doi.org/10.1111/icad.12408
- [11]Vasco Elbrecht and Florian Leese. Can DNA-based ecosystem assessments quantify species abundance? Testing primer bias and biomass—sequence relationships with an innovative metabarcoding protocol. PLOS ONE, 10(7):e0130324, July 2015. doi: 10.1371/journal.pone.0130324. 3, 18DOI
- [12]Vasco Elbrecht and Florian Leese. Validation and development of COI metabarcoding primers for freshwater macroinvertebrate bioassessment. Frontiers in Environmental Science, 5(11), 10 April 2017. DOI: 10.3389/fenvs.2017.00011. 3DOI
- [13]Vasco Elbrecht, Thomas W. A. Braukmann, Natalia V. Ivanova, Sean W. J. Prosser, Mehrdad Hajibabaei, Michael Wright, Evgeny V. Zakharov, Paul D. N. Hebert, and Dirk Steinke. Validation of COI metabarcoding primers for terrestrial arthropods. PeerJ, 7:e7745, 7 October 2019. doi: 10.7717/peerj.7745. 3, 19DOI
- [14]Furneaux, B., Roslin, T., Hardwick, B., Kedraon, D., Autto, H., Banelyte, G., deWaard, J. R., deWaard, S. L., Farrell, A., Kaltopää, O., Kristensen, E., Rogers, H. M., Rogers, J. E., Sones, E. V., Zakharov, E., & Ovaskainen, O. (2026). Is there a fly in my soup? To what extent do metabarcoding and individual barcoding tell the same story?. Molecular Ecology Resources, 26(7), e70195. https://doi.org/10.1111/1755-0998.70195
- [15]Tiancheng Gao and Graham W. Taylor. BarcodeMamba: State space models for biodiversity analysis. arXiv preprint arXiv:2412.11084, 2024. DOI: 10.48550/arXiv.2412.11084. 2, 6DOI
- [16]Tiancheng Gao, Scott C. Lowe, Brendan Furneaux, Angel X. Chang, and Graham W. Taylor. BarcodeMamba+: Supervised advances for state-space models in fungal biodiversity research. In NeurIPS 2025 Workshop for Imageomics: Discovering Biological Knowledge from Images Using AI, 2025. URL https://openreview.net/forum?id=wOgcLKq1Hg. 6openreview.net/forum?id=wOgcLKq1Hg
- [17]Matthias F. Geiger, Jerome Moriniere, Axel Hausmann, Gerhard Haszprunar, Wolfgang Wägele, Paul D. N. Hebert, and Björn Rulik. Testing the global malaise trap program – how well does the current barcode reference library identify flying insects in Germany? Biodiversity Data Journal, 4:e10671, December 2016. ISSN 1314-2836. doi: 10.3897/BDJ.4.e10671. 3, 17DOI
- [19]Caspar A. Hallmann, Martin Sorg, Eelke Jongejans, Henk Siepel, Nick Hofland, Heinz Schwan, Werner Stenmans, Andreas Müller, Hubert Sumser, Thomas Hörren, Dave Goulson, and Hans de Kroon. More than 75 percent decline over 27 years in total flying insect biomass in protected areas. PLOS ONE, 12(10):e0185809, 18 October 2017. doi: 10.1371/journal.pone.0185809. 2DOI
- [21]Harrison, J. G., Calder, W. J., Shuman, B., & Buerkle, C. A. (2021). The quest for absolute abundance: the use of internal standards for DNA-based community ecology. Molecular Ecology Resources, 21(1), 30–43. https://doi.org/10.1111/1755-0998.13247
- [22]Ela Iwaszkiewicz-Eggebrecht, Emma Granqvist, Karol H. Nowak, Catalina Valdivia, Mateusz Buczek, Amrita Srivathsan, Emily Hartop, Andreia Miraldo, Tomas Roslin, Ayco J. M. Tack, Piotr Łukasik, Rudolf Meier, and Fredrik Ronquist. Accuracy of occurrence and abundance estimates from insect metabarcoding. Methods in Ecology and Evolution, 17(8):2343–2363, 2026. doi: 10.1111/2041-210x.70351. 3, 4, 9DOI
- [23]Yinqiu Ji, Louise Ashton, Scott M. Pedley, David P. Edwards, Yong Tang, Akihiro Nakamura, Roger Kitching, Paul M. Dolman, Paul Woodcock, Felicity A. Edwards, Trond H. Larsen, Wayne W. Hsu, Suzan Benedick, Keith C. Hamer, David S. Wilcove, Catharine Bruce, Xiaoyang Wang, Taal Levi, Martin Lott, Brent C. Emerson, and Douglas W. Yu. Reliable, verifiable and efficient monitoring of biodiversity via metabarcoding. Ecology Letters, 16(10):1245–1257, October 2013. doi: 10.1111/ele.12162. 2, 3DOI
- [24]Yinqiu Ji, Tea Huotari, Tomas Roslin, Niels Martin Schmidt, Jiaxin Wang, Douglas W. Yu, and Otso Ovaskainen. SPIKEPIPE: A metagenomic pipeline for the accurate quantification of eukaryotic species occurrences and intraspecific abundance change using DNA barcodes or mitogenomes. Molecular Ecology Resources, 20(1):256–267, 2020. doi: 10.1111/1755-0998.13057. 3, 9DOI
- [25]Dave Karlsson, Emily Hartop, Mattias Forshage, Mathias Jaschhof, and Fredrik Ronquist. The Swedish malaise trap project: A 15 year retrospective on a countrywide insect inventory. Biodiversity Data Journal, 8:e47255, January 2020. doi: 10.3897/bdj.8.e47255. 2DOI
- [26]Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment anything in high quality. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 29914–29934. Curran Associates, Inc., 2023. DOI: 10.52202/075280-1303. 21DOI
- [27]Konstantin Klemmer, Esther Rolf, Caleb Robinson, Lester Mackey, and Marc Rußwurm. SatCLIP: Global, general-purpose location embeddings with satellite imagery. Proceedings of the AAAI Conference on Artificial Intelligence, 39(4):4347–4355, 11 April 2025. DOI: 10.1609/aaai.v39i4.32457. 6, 9DOI
- [28]Philip D. Lamb, Ewan Hunter, John K. Pinnegar, Simon Creer, Richard G. Davies, and Martin I. Taylor. How quantitative is metabarcoding: A meta-analytical approach. Molecular Ecology, 28(2):420–430, January 2019. DOI: 10.1111/mec.14920. 3DOI
- [29]Joanne E. Littlefair, Lauren D. Hayhurst, Matthew C. Yates, Michael D. Rennie, and Melania E. Cristescu. Environmental DNA metabarcoding reflects rank-based abundance of fish. Journal of Applied Ecology, 63(3):e70328, March 2026. DOI: 10.1111/1365-2664.70328. 3DOI
- [30]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), *Computer Vision – ECCV 2024*, pp. 38–55. Springer Nature Switzerland, 2025. doi: 10.1007/978-3-031-72970-6_3. 21DOI
- [31]Silvia Matesanz, David S. Pescador, Beatriz Pías, Ana M. Sánchez, Julia Chacón-Labella, Angela Illuminati, Marcelino de la Cruz, Jesús López-Angulo, Neus Marí-Mena, Antón Vizcaíno, and Adrián Escudero. Estimating belowground plant abundance with DNA metabarcoding. *Molecular Ecology Resources*, 19(5):1265–1277, September 2019. doi: 10.1111/1755-0998.13049. 3DOI
- [32]Michael R. McLaren, Amy D. Willis, and Benjamin J. Callahan. Consistent and correctable bias in metagenomic sequencing experiments. *eLife*, 8:e46923, 10 September 2019. doi: 10.7554/eLife.46923. 2, 4, 5, 7, 28DOI
- [33]Pablo Millan Arias, Niousha Sadjadi, Monireh Safari, Zeming Gong, Austin T. Wang, Joakim Bruslund Haurum, Iuliia Zarubiieva, Dirk Steinke, Lila Kari, Angel X. Chang, Scott C. Lowe, and Graham W. Taylor. BarcodeBERT: transformers for biodiversity analyses. *Bioinformatics Advances*, 6(1):vbag054, 19 February 2026. doi: 10.1093/bioadv/vbag054. 2, 5, 22DOI
- [34]Johanna Orsholm, John Quinto, Hannu Autto, Gaia Banelyte, Nicolas Chazot, Jeremy deWaard, Stephanie deWaard, Arielle Farrell, Brendan Furneaux, Bess Hardwick, Nao Ito, Amlan Kar, Oula Kalttopää, Deirdre Kerdraon, Erik Kristensen, Jaclyn McKeown, Tommi Mononen, Ellen Nein, Hanna Rogers, Tomas Roslin, Paula Schmitz, Jayme Sones, Maija Sujala, Amy Thompson, Evgeny V. Zakharov, Iuliia Zarubiieva, Akshita Gupta, Scott C. Lowe, and Graham W. Taylor. A multi-modal dataset for insect biodiversity with imagery and DNA at the trap and individual level. *Scientific Data*, 13(1):630, April 2026. doi: 10.1038/s41597-026-07251-x. 3DOI
- [35]Sameer Padhye, Liliana Ballesteros-Mejia, and Sujeevan Ratnasingham. *BOLDconnectR: Retrieve, Transform and Analyze the Barcode of Life Data Systems Data*, 2025. URL https://CRAN.R-project.org/package=BOLDconnectR. R package version 1.0.0. 17, 20DOI
- [36]Pagès, H., Aboyoun, P., Gentleman, R., & DebRoy, S. (2024). Biostrings: Efficient manipulation of biological strings. https://bioconductor.org/packages/Biostringsbioconductor.org/packages/Biostrings
- [37]M. F. Polz and C. M. Cavanaugh. Bias in template-to-product ratios in multitemplate PCR. *Applied and Environmental Microbiology*, 64(10):3724–3730, 1 October 1998. doi: 10.1128/AEM.64.10.3724-3730.1998. 17, 18DOI
- [38]S. W. J. Prosser, R. M. Floyd, K. A. Thompson, S. K. Monckton, and P. D. N. Hebert. BOLDistilled: Automated construction of comprehensive but compact DNA barcode reference libraries. *Molecular Ecology Resources*, 25(8):e70043, November 2025. doi: 10.1111/1755-0998.70043. 17DOI
- [39]Sean W. J. Prosser, Nicholas W. Bard, Ken A. Thompson, Robin M. Floyd, Sameer Padhye, Emine Ozsahin, Saeideh Jafarpour, and Paul D. N. Hebert. The metabarcoding analysis pipeline (MAP): Simple, accurate, and flexible metabarcoding. *bioRxiv*, pp. 2026.07.22.740107, 23 July 2026. doi: 10.64898/2026.07.22.740107. 3, 17DOI
- [40]Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. In *Forty-third International Conference on Machine Learning*, 2026. URL https://openreview.net/forum?id=SxsyLjJfWB. 8, 26, 30openreview.net/forum?id=SxsyLjJfWB
- [41]Sujeevan Ratnasingham and Paul D. N. Hebert. A DNA-based registry for all animal species: the barcode index number (BIN) system. *PLOS ONE*, 8(7):e66213, 8 July 2013. doi: 10.1371/journal.pone.0066213. 2DOI
- [42]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In *International Conference on Learning Representations*, pp. 2805–2812, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/45c1f6a8cbf2da59ebf2c802b4f742cd-Abstract-Conference.html. 21proceedings.iclr.cc/paper_files/paper/2025/hash/45c1f6a8cbf2da59ebf2c802b4f742cd-Abstract-Conference.html
- [43]Safari, M., Millan Arias, P., Lowe, S. C., Kari, L., Chang, A. X., & Taylor, G. W. (2025). Enhancing DNA foundation models to address masking inefficiencies. ICLR 2025 Workshop on AI for Nucleic Acids. https://doi.org/10.48550/arxiv.2502.18405
- [44]B. E. Sample, R. J. Cooper, R. D. Greer, and R. C. Whitmore. Estimation of insect biomass by length and width. The American Midland Naturalist, 129(2):234–240, 1 April 1993. doi: 10.2307/2426503. 22DOI
- [45]Eric W. Sayers, Mark Cavanaugh, Linda Frisse, Kim D. Pruitt, Valerie A. Schneider, Beverly A. Underwood, Linda Yankie, and Ilene Karsch-Mizrachi. GenBank 2025 update. Nucleic Acids Research, 53(D1):D56–D61, January 2025. doi: 10.1093/nar/gkae1114. 19DOI
- [46]Stefan Schneider, Graham W. Taylor, Stefan C. Kremer, Patrick Burgess, Jillian McGroarty, Kyomi Mitsui, Alex Zhuang, Jeremy R. deWaard, and John M. Fryxell. Bulk arthropod abundance, biomass and diversity estimation using deep learning for computer vision. Methods in Ecology and Evolution, 13(2):346–357, 14 December 2021. doi: 10.1111/2041-210X.13769. 21DOI
- [47]Ida Bærholm Schnell, Kristine Bohmann, and M. Thomas P. Gilbert. Tag jumps illuminated – reducing sequence-to-sample misidentifications in metabarcoding studies. Molecular Ecology Resources, 15(6):1289–1303, 1 November 2015. doi: 10.1111/1755-0998.12402. 5DOI
- [48]Wiebke Sickel, Vera Zizka, Alice Scherges, Sarah J. Bourlat, and Petra Dieker. Abundance estimation with DNA metabarcoding – recent advancements for terrestrial arthropods. Metabarcoding and Metagenomics, 7:e112290, November 2023. doi: 10.3897/mbmg.7.112290. 3DOI
- [49]M. T. Suzuki and S. J. Giovannoni. Bias caused by template annealing in the amplification of mixtures of 16S rRNA genes by PCR. Applied and Environmental Microbiology, 62(2):625–630, February 1996. doi: 10.1128/aem.62.2.625-630.1996. 5, 17, 19DOI
- [50]Roel van Klink, Diana E. Bowler, Konstantin B. Gongalsky, Ann B. Swengel, Alessandro Gentile, and Jonathan M. Chase. Meta-analysis reveals declines in terrestrial but increases in freshwater insect abundances. Science, 368(6489):417–420, 24 April 2020. doi: 10.1126/science.aax9931. 2DOI
- [51]David L. Wagner, Eliza M. Grames, Matthew L. Forister, May R. Berenbaum, and David Stopak. Insect decline in the anthropocene: Death by a thousand cuts. Proceedings of the National Academy of Sciences, 118(2):e2023989118, January 2021. doi: 10.1073/pnas.2023989118. 2DOI
- [52]Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. SegGPT: Towards segmenting everything in context. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1130–1140, 2023. doi: 10.1109/ICCV51070.2023.00110. URL https://openaccess.thecvf.com/content/ICCV2023/html/Wang_SegGPT_Towards_Segmenting_Everything_in_Context_ICCV_2023_paper.html. 21DOI
- [53]Wickham, H., & Bryan, J. (2025). readxl: Read Excel Files. <a href="https://doi.org/https://CRAN.R-project.org/package=readxl" rel="noopener noreferrer" target="_blank">https://doi.org/https://CRAN.R-project.org/package=readxl</a>
- [54]Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Milton Bache, S., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., … Yutani, H. (2019). Welcome to the Tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686
- [55]David Winter. rentrez: An R package for the NCBI eUtils API. PeerJ Preprints, 5:e3179v2, 2017. DOI: 10.7287/peerj.preprints.3179v2. 19DOI
- [56]Erik S. Wright. Fast and flexible search for homologous biological sequences with DECIPHER v3. The R Journal, 16(2):191–200, 2024. DOI: 10.32614/RJ-2024-022. 19DOI
- [57]Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. Do current multi-task optimization methods in deep learning even help? In Advances in Neural Information Processing Systems, volume 35, pp. 13597–13609. Curran Associates, Inc., 2022. DOI: 10.52202/068431-0988. 30DOI
- [58]Louie H. Yang and Claudio Gratton. Insects as drivers of ecosystem processes. Current Opinion in Insect Science, 2:26–32, August 2014. DOI: 10.1016/j.cois.2014.06.004. 2DOI
- [59]Douglas W. Yu, Yinqiu Ji, Brent C. Emerson, Xiaoyang Wang, Chengxi Ye, Chunyan Yang, and Zhaoli Ding. Biodiversity soup: metabarcoding of arthropods for rapid biodiversity assessment and biomonitoring. Methods in Ecology and Evolution, 3(4):613–623, August 2012. DOI: 10.1111/j.2041-210X.2012.00198.x. 2DOI
- [60]Lucie Zinger, Aurélie Bonin, Inger G. Alsos, Miklós Bálint, Holly Bik, Frédéric Boyer, Anthony A. Chariton, Simon Creer, Eric Coissac, Bruce E. Deagle, Marta De Barba, Ian A. Dickie, Alex J. Dumbrell, Gentile Francesco Ficetola, Noah Fierer, Luca Fumagalli, M. Thomas P. Gilbert, Simon Jarman, Ari Jumpponen, Håvard Kauserud, Ludovic Orlando, Johan Pansu, Jan Pawlowski, Leho Tedersoo, Philip Francis Thomsen, Eske Willerslev, and Pierre Taberlet. DNA metabarcoding—need for robust experimental designs to draw sound ecological conclusions. Molecular Ecology, 28(8):1857–1862, 1 April 2019. doi: 10.1111/mec.15060. 5DOI








