betaFeedback

WorksComputer Science

How Finely Can Vision-Language Models See Biology? Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning

Abstract

In biodiversity monitoring, a single image raises several questions: which taxon, how many, which trait, and whether the individual has been seen before. Vision-language models (VLMs) could answer all of these from the image, but their ability is largely untested in biology, as existing benchmarks focus mostly on species identification within a few taxonomic groups. We present Bio-VQA, the largest biological visual question answering (VQA) benchmark and the only one to span four aspects of biological visual understanding: hierarchical taxonomic reasoning, fine-grained attribute grounding, quantitative spatial reasoning, and individual re-identification. Bio-VQA contains 1.86M questions over 217K images from 11 public datasets, and extends right-or-wrong scoring to graded scoring, so a near-miss counts for more than a distant one. Evaluating 18 open-weight models from 3.75B to 38.4B parameters, we find that general visual capabilities do not transfer to biological organisms: the best model counts animals exactly on 41.7% of images with 2 to 10 individuals, less than half its accuracy on everyday objects at the same counts. Taxonomic accuracy falls to one sixth, from 0.72 at phylum to 0.12 at scientific name, and scaling helps mostly at coarse ranks. The gap to a domain specialist widens accordingly, from 0.731 against BioCLIP-2's 0.874 at class (1.2x) to 0.269 against 0.544 at scientific name (2x). On trait size measurement, 15 of the 18 models do no better than an image-blind baseline that predicts one fixed number for every specimen, and on re-identification, 17 of the 18 models fall below chance under adversarial conditions. These failures persist across a tenfold range of model size. We hope Bio-VQA becomes a standard test of how well VLM capabilities transfer to the biological domain.

Introduction

Images are the primary record for modern biodiversity monitoring [10, 13, 23], but the large volume of data from camera traps and digitization efforts has created an annotation bottleneck that specialist taxonomic models and experts cannot clear [46, 49, 53]. VLMs solve this by answering questions from images, and are being applied across domains like medical imaging, autonomous driving and document understanding. Yet we lack the benchmarks to measure how well VLMs answer biological questions [11, 33, 44, 45]. Current evaluations fall short in four ways: Narrow taxonomic coverage. They restrict themselves to a few animal groups. VLM4Bio [32], the closest prior work in scope, draws its questions from three animal groups, while others focus only on crop species [47, 57] or a single habitat at a small scale [65]. Narrow question scope. They focus heavily on basic taxonomic naming, ignoring major biological aspects like morphological traits, spatial quantification or re-identification. Fixed scoring. They mark an answer either right or wrong, which penalizes a near-miss in the correct genus exactly the same as a complete miss in the wrong kingdom. Weaker baselines. They report random chance for multiple-choice items, and compares one task against a specialist [32]. Neither bound is available per task, and random chance is weaker than an image-blind predictor that exploits answer distribution. A weaker baseline understates the floor and inflates the models’ scores. These shortfalls leave a large part of VLMs’ biological visual reasoning capabilities unmeasured.

In Bio-VQA, we address the gaps with 24 tasks across four components, graded scoring, and a baseline for each task. The components are: hierarchical taxonomic reasoning (C1), fine-grained attribute grounding (C2), quantitative spatial reasoning (C3), individual re-identification (C4). C1 probes eight ranks across 10 organismal groups drawn from TreeOfLife-200M [18]. C2 tests visual traits, attributes and anatomic localization on CUB-200-2011 [55]. C3 decomposes spatial understanding into atomic tasks. C4 tests whether two photographs show the same individual. Our contributions are:

  1. The Benchmark: Bio-VQA is the largest biological VQA dataset to date, with 1.86M questions across 217K images, and we uniquely combine taxonomic, morphological, spatial, and re-identification tasks.

  2. Biology-Aware Metrics: Besides strict exact matches, we report taxonomy-aware soft match and tree-distance scoring for C1; chance-adjusted graded score for C2, validated against three human annotators and an 11-configuration sensitivity sweep; bounded count agreement for C3; and signal-detection scoring for C4.

  3. Rigorous Baselines: We bound every task below by a computed image-blind floor, or above by a measured specialist ceiling, placing generalist performance between guessing and genuine domain expertise.

  4. Cross-Component Analysis: We run correlation study across all components to test whether a model’s ability on one transfers to another. For the strongest models it does, lower down the rank, it does not. The results from 18 open-weight VLMs reveal a domain gap. VLMs handle coarse categories well but fail on fine-grained tasks, and within the models we test, larger models mostly improve at coarse ranks they already know. Outside taxonomy, they fail to count conspecifics exactly or to distinguish individuals when appearance is controlled. In short, biological imagery requires a strict granularity of discrimination that

nation that general pretraining corpora do not reward.

Related Work

General VLM benchmarks. Multi-task suites like MM-Bench [28], SEED-Bench [24], BLINK [12], MMVP [51], and CV-Bench [50] evaluate VLMs by breaking overall accuracy into per-ability scores (e.g. counting, spatial relations, depth, OCR), and AVA-Bench [31] extends the decomposition to 14 atomic visual abilities (e.g. absolute distance, relative distance). These suites establish that decomposing one score into per-ability components exposes failures that an aggregate score would mask. Biological organisms appear in these suites, but incidentally rather than as a focus. So we adopt their per-ability component decomposition and apply it to a structured set of biological tasks.

Biological VQA benchmarks. The closest reference of our work is VLM4Bio [32]. It evaluates 12 models on 469K QA pairs across 30K images of fishes, birds, and butterflies. We extend this line of work along three axes, not just in scale. First, we cover ten organismal groups, testing underrepresented branches like arachnids and bryophytes. Second, we apply open-ended and multiple-choice questions on the exact same items to measure the recognition-recall gap. Finally, we explore tasks on spatial reasoning and individual re-identification, which are untested in prior biological VQA benchmarks. Other domain benchmarks are also limited in scope: J-EDI QA [65] focuses on 100 deep-sea images, PlantExpertVQA [47] is limited to crop diseases, and Agro-Cot [57] targets agricultural chain-of-thought reasoning. Other biodiversity evaluations either lack imagery [52] or focus on retrieval rather than VQA [53]. Table 1 compares Bio-VQA against these prior works.

BenchmarkQAImg.Org. grp.TaxAttSpaInd
General
MMBench [28]3.0K3.0K—✗○○✗
BLINK [12]3.8K7.4K—✗○✓✗
CV-Bench [50]2.6K2.6K—✗○✓✗
AVA-Bench [31]218K218K—✗✓✓✗
Biology
J-EDI QA [65]0.1K0.1K1✓✓✗✗
PlantExpertVQA [47]765K151K1○✓✗✗
VLM4Bio [32]469K30K3✓✓✗✗
Bio-VQA (ours)1.86M217K10✓✓✓✓

Table 1. Comparison with prior benchmarks. Columns report the number of question-answer pairs (QA), images (Img.) and organismal groups (Org. grp.); and whether they cover question types: taxonomic (Tax), attribute (Att), spatial (Spa) and individual identity (Ind). ✓ means the question type is a primary task family, ○ that it appears incidentally, and ✗ that it is absent.

Domain specialists. For taxonomic identification, contrastive biological encoders are the strongest references:

BioCLIP [49] and BioCLIP-2 [18] are trained at tree-of-life scale, and we use the latter as the specialist baseline for C1. Individual re-identification has a longer, separate line of work, from hand-crafted local features [9] to learned embeddings [20, 35, 37] and multi-species descriptors [6, 40]; all are purpose-built, and no survey of the area [5] reports VLMs. C4 asks whether VLMs can re-identify individuals under controlled difficulty levels, scored with signal detection theory [17, 30] against a specialist, MegaDescriptor.

Evaluation methodology. Exact-match scoring penalizes near-misses (such as identifying correct genus with wrong species) identically to complete failures (wrong family). Hierarchical scoring with Wu–Palmer [63] and Lin [25] similarity is standard in taxonomy-aware evaluation [16], and chance correction via Cohen’s κ\kappa [8] is standard in annotation agreement. However, prior VQA benchmarks do not combine these. In Bio-VQA, we combine them and report every graded metric along with the exact match.

Curation of Bio-VQA

Design principles. We curate Bio-VQA on three principles. First, we base our questions on actual biological workflows: naming organisms, describing traits, counting and measuring specimens, and re-identifying individuals. Second, we generate a question only where an existing annotation supplies the ground truth, so every answer can be checked against its source. Third, within each component, we stratify questions into difficulty tiers along an axis that component defines (e.g. common vs rare taxa on C1). The four subsections below give the reasoning and the construction for each component.

C1: Hierarchical taxonomic reasoning

Taxonomic identification is the most common biological visual task, and contrastive encoders like BioCLIPs [18, 49] and BioTrove-CLIP [64] are specialists at this. We ask how

far a generalist VLM stands from the specialists on the same task. Current works evaluate species identification on a limited taxa [32, 47]. We extend this along two axes: (a) we ask the question at eight ranks rather than at species alone, (b) we ask it of ten organismal groups; six of which do not appear in prior works.

We sample a 100K subset of TreeOfLife-200M [18] spanning 626 families, 2311 genera (6×6\times VLM4Bio) and 3403 species (4×4\times VLM4Bio). The source is long-tailed, so a random sample would be dominated by a few abundant taxa. We therefore select 10K images from each organismal group, on three constraints: (a) complete lineage from phylum to species and common names, so that no question is generated without a ground truth; (b) a minimum resolution threshold; (c) a per-species cap, to avoid abundant species within a group. Then, we ask each image for its phylum, class, order, family, genus, species, scientific name and common name1 (Figure 2). We test two framings on the common name questions, open-ended and four-way multiple choice, on the same image and same target. This tests what a model recalls from what it recognizes once given the options, and evaluate the recognition-recall gap. For the multiple choice, we draw the three distractors uniformly from the taxa of the same class as the target, at whichever rank the question asks.

Taxonomic identification question for a jumping spider

Figure 2. C1 asks one image for its identity at eight ranks, from phylum down to common name, and repeats common name as four-way multiple choice. Species is the epithet and scientific name is the binomial, so we score the two separately.

Then we assign difficulty tier labels on questions by how common the target taxon is in TreeOfLife-200M. At each rank, we take the top third of taxa by image count as Easy, the bottom third as Hard, and the rest as Medium; so the tiers separate taxa a model has seen often from ones it has barely seen. Finally, we address two properties of the source labels. The GBIF vernacular records are not filtered by language, and contains Swedish, Danish and Norwegian names in the common-name field. A model answering with the correct English name would be scored wrong, so we resolve every non-English common name to English against the iNaturalist and Wikidata APIs [23, 58]. The second is synonymy: a cougar answered as Felis concolor is the same animal as Puma concolor. We build a synonym map from GBIF Backbone [13] and accept any synonym of the target.

C2: Fine-grained attribute grounding

Fine-grained traits play a key role in trait-based ecology, and recognizing them differs from naming a taxon. In C2, we evaluate how well VLMs recognize these fine-grained visual attributes. We use CUB-200-2011 [55], a well-known bird dataset that provides 312 binary attributes per image, each named as a part, a property, and a value (has_wing_color::brown). Evaluating a model on 312 sparse yes/no questions is uninformative, because most flags are absent and a constant predictor that answers no scores well. So we invert the mapping: we combine a part and a property (wing color) into a single question and use the marked value as the ground truth (brown).

This gives us four types of questions: morphological attributes, relative salience, anatomical localization and naming (Figure 3). For multiple choice, we draw distractors from the other values attested for that same part and property, and exclude any value that is itself true of the image, so every option is a real trait of some bird and exactly one is correct. A model must therefore ground its answer rather than use the process of elimination. We use the annotator certainty on each label to select and filter candidate traits, and report that handling in the appendix. For anatomical localization, we render a marker onto the image and ask which body part it is, and add distractors using two controls: one distractor shares the marker’s color but not its shape, another shares its shape but not its color. Neither cue identifies the marker alone, so a model must bind color and shape together to answer. Because we render all the 11 788 source images, the component comprises 23 576 images and 824 104 questions.

Four categories of questions in C2

Figure 3. C2 poses four categories of question, with one example item each. We ask naming both open-ended and as multiple choice on the same image. Anatomical localization is visually prompted, so the question refers to a rendered marker rather than to a named part.

C3: Quantitative spatial reasoning

Spatial reasoning is widely studied in general VQA benchmarks [2, 7, 12, 26, 41], and it also covers the most common operations in biodiversity fieldwork. In C3, we ask how well VLMs perform these tasks on biological subjects. Following the task-decomposition approach of AVA-Bench [31], we break spatial reasoning into seven tasks (Figure 4). The decomposition attributes a failure to a specific ability rather than to spatial reasoning in general. We add optical character recognition (OCR) as an eighth task and a negative control. Locating the label is a spatial task, but the answer depends on transcribing characters.

C3’s eight spatial tasks, with one example item each, its answer format and its ground truth

Figure 4. C3’s eight spatial tasks, with one example item each, its answer format and its ground truth. The first panel asks counting, OCR and relative position of the same beetle plate. The relative distance and counting tasks are multiple-choice, rest are open-ended.

We use five ecological datasets (Table 2). We select these for multi-organism images, which tasks like counting and relative position require, and which prior biological VQA works do not hold [32, 47]. We aggregate them into ten 500-image subsets, on two constraints. First, we sample

ComponentTasksDataImagesQuestions
C1Hierarchical taxonomic reasoning91100,000890,000
C2Fine-grained attribute grounding4123,576824,104
C3Quantitative spatial reasoning88{sup}†`78,139132,504
C4Individual re-identification3515,00015,000
Total2411{sup}†`216,7151,861,608

Table 2. Bio-VQA statistics: We draw C1 from TreeOfLife-200M [18]; C2 from CUB-200-2011 [55]; C3 from WAID [36], Bees [4], 2018-NEON-Beetles [22], DePondFi [54], GZCD [38]; and C4 from WhaleSharkID [59], BelugaID [60], HyenaID [61], LeopardID [62], GZCD [38]. †C3 uses WhaleSharkID, HyenaID, LeopardID, GZCD on two tasks: so the total count becomes 11.

using count stratification for a balanced mix of sparse and dense scenes. Second, we discard images whose annotation completeness we cannot verify, since a partial annotation penalizes a correct count.

Next, we evaluate each task only on subsets where the ground truth exists. Absolute and relative distance run on the beetle dataset, as that is the only one with recorded trait and scalebar measurements. Relative position also runs on beetles, because the specimens lie flat in a lattice at one focal depth, so top-right and left are genuine two-dimensional relations and not perspective artifacts. Localization and viewpoint need a single dominant subject, otherwise a model has to locate the animal before it can answer and the tasks stop being isolated. So we leverage the wildlife catalogs from C4 (§ 3.4), where annotations explicitly record the bounding box and the viewing angle. In total, C3 holds 132 504 questions over 78 139 images across eight tasks.

C4: Individual re-identification

Animal re-identification is a highly specialized task, handled by purpose-built matching models [6, 9, 40] and field experts. We are not aware of a benchmark that asks it of a generalist VLM. We are interested to know if VLM focuses on identity (e.g., zebra’s stripe pattern, leopard’s spots) or appearance similarity, on re-id tasks. We test this with a minimal probe. We place two images side by side and ask whether these are the same individual or not (Figure 1d).

Bio-VQA benchmark components

Figure 1. Bio-VQA is a benchmark on biological domain, consisting of four components: (a) hierarchical taxonomic reasoning, (b) fine-grained attribute grounding, (c) quantitative spatial reasoning (3 of 8 tasks shown: counting, OCR, relative position), (d) re-identification.

We build pairs from five wildlife catalogs (Table 2), and set their difficulty using DINOv2 [39] embeddings. Easy pairs an animal with its nearest match, often frames taken seconds apart, and pairs different animals that look nothing alike, so appearance and identity agree. Random draws both without using appearance, so appearance carries no signal. Hard reverses easy: different individuals are drawn from each other’s nearest neighbors, and the same individual from its furthest, so appearance and identity disagree. We retrieve within viewpoint groups, since viewpoint dominates the embedding space. Each catalog contributes 1000 balanced pairs per condition, and we sample same-individual pairs first, since the catalogs hold fewer repeat sightings than distinct animals. The hard condition is adversarial by construction. In these pairs the two animals share same pose, lighting, background and coat tone. They differ only in the markings that identify them. Therefore a model that decides by appearance yields the wrong answer.

Evaluation Pipeline

Models and Decoding

We evaluate 18 open-weight models from 3.75B to 38.4B parameters: Qwen2.5/3-VL [42, 43], InternVL3/3.5 [56, 66], Gemma-3/4 [14, 15], Molmo 2 [3], Pixtral [34], Phi-3.5-Vision [1], Llama-3.2-Vision [29], and two LLaVA-NeXT variants [27]. Two 30B models are mixture-of-expert variants that activate 3B parameters per token. All run under one protocol: greedy decoding, the same question and answer-format instruction for every model within a question family [48], no chain-of-thought, one attempt per item, and one scorer per component. We report parse and letter-text mismatch rates to separate formatting failures from knowledge errors [28]. The format instructions show an example answer; the supplement lists the exact strings.

Scoring

Prior biological VQA scores by strict match [32, 47, 65], which credits a near miss and a random guess alike. We add one graded metric per component.

C1: quantifying taxonomic error. On a warbler, exact match scores Turdidae and Fungi alike, although one is near miss and one is wrong by far. To partially credit the near miss, we measure taxonomic distance. With ℓ=LCA⁡(y^,y)\ell= \operatorname{LCA}(\hat{y},y), the deepest clade containing both answers, Wu–Palmer similarity [63] scores the depth of that shared ancestor:

wup⁡(y^,y)=2depth⁡(ℓ)depth⁡(y^)+depth⁡(y)(1)\operatorname{wup}(\hat{y},y) = \frac{2\operatorname{depth}(\ell)}{\operatorname{depth}(\hat{y})+\operatorname{depth}(y)} \tag*{(1)}

We also report soft exact match, which accepts any synonym of the gold answer from the accepted-name relations of the GBIF Backbone [13], and give Lin similarity [25] and severity in the appendix.

C2: near misses and uneven answer spaces. On a brown wing, red is closer than green, and the option count runs from 3 for bill length to 15 for color, so a raw graded score confounds ability with option count. Our Chance-Adjusted Graded Score (CAGS) corrects both. Let s(y^,y)∈[0,1]s(\hat{y},y) \in[0,1] be the similarity between two answers and E0[s]\mathbb{E}_0[s] its value under a random guess:

CAGS⁡=E[s(y^,y)]−E0[s]1−E0[s](2)\operatorname{CAGS} = \frac{\mathbb{E}[s(\hat{y},y)]-\mathbb{E}_0[s]}{1-\mathbb{E}_0[s]} \tag*{(2)}

CAGS is 0 at chance and 1 at perfect for any option count. We take ss from CIEDE2000 distance for colors and annotation co-occurrence for parts, and validate it in § 5.2.

C3: grading error and establishing baselines. Predicting 14 beetles for 15 beats predicting 20, so we score counting by relative error, macro-averaged over corpora whose count distributions differ:

CAS⁡(n^,n)=max⁡(0,1−∣n^−n∣n)(3)\operatorname{CAS}(\hat{n},n) = \max\left(0,1-\frac{|\hat{n}-n|}{n}\right) \tag*{(3)}

Localization uses mean IoU and the categorical tasks balanced accuracy, with macro-averaged agreement for viewpoint. Every task also carries a floor ϕ\phi, the best score an image-blind rule reaches over the modal count, a per-type constant and majority class. We report lift (S−ϕ)/(1−ϕ)(S-\phi)/(1-\phi), which is 0 when a model matches that rule and negative when it does worse.

C4: same or not same. C4 is a verification task [21]: balanced same and not-same pairs, scored by accuracy. A VLM returns a decision rather than a similarity score, so it sits at one operating point where accuracy conflates discrimination with bias, and answering yes to every pair scores 50%. We separate the two using signal detection theory [17, 30], writing HH for the yes rate on same pairs and FF for the rate on different ones:

d′=z(H)−z(F),c=−12[z(H)+z(F)](4)d' = z(H) - z(F), \qquad c = -\frac{1}{2}\left[z(H) + z(F)\right] \tag*{(4)}

where zz is the inverse normal CDF, d′d' is discriminability and cc is model bias. We apply the log-linear correction first, since zz diverges when HH or FF reaches 0 or 1 [19], and report 95% bootstrap intervals for accuracy over 2000 resamples per cell. Balanced accuracy, Matthews correlation and Cohen’s κ\kappa are reported in the appendix.

Experiment Results

Table 3 reports all 18 models on 24 tasks across 4 components. We read it against two reference points. The floor row gives the score an image-blind rule reaches. The two specialist rows give the score a purpose-built model reaches.

#Modelphyclsordfamgenspcscicomsalmrplocnamcntrcntadstrdstlclrposviewocrd′d'acc
BioCLIP-2 (tol)∗^{*}.925.874.821.775.691.545.544.500————————————.733.683
MegaDescriptor∗∗^{**}——————————————————————
1Qwen3-VL-32B.831.615.712.612.444.066.244.078.584.644.475.731.789.555.622.443.866.550.427.900.696.636
2Qwen3-VL-30B-A3B.864.731.757.612.417.072.221.107.558.631.453.658.795.483.528.423.861.355.391.979.518.602
3InternVL3-38B.911.683.638.484.355.054.189.044.562.645.413.669.799.567.367.363.744.205.373.105.463.581
4Gemma-4-12B.880.686.578.363.183.021.065.043.592.621.431.374.753.563.385.443.589.495.331.345.351.567
5Qwen3-VL-8B.887.680.720.589.406.046.202.065.531.624.441.643.739.414.303.345.862.421.373.711.492.572
6InternVL3.5-38B.746.636.624.363.271.038.128.025.559.638.375.536.765.504.503.360.412.219.340.192.258.542
7Molmo2-8B.823.563.469.286.137.021.038.016.503.643.474.330.623.416.290.358.055.457.483.692.491.578
8Qwen2.5-VL-7B.763.504.673.541.420.047.269.062.541.580.356.713.724.410.334.337.308.213.395.916.442.541
9InternVL3.5-30B-A3B.486.469.545.310.240.034.118.024.537.609.350.539.727.489.378.333.289.252.309.188.568.612
10InternVL3-8B.827.497.478.322.262.032.124.020.508.584.346.581.738.320.297.304.246.203.335.314.684.601
11InternVL3.5-8B.819.531.421.266.191.023.077.015.495.603.372.457.747.408.263.299.236.224.264.460.236.544
12Gemma-3-12B.790.688.626.429.225.048.093.047.466.551.410.511.606.415.352.280.329.145.235.352.120.505
13Qwen2.5-VL-3B.416.457.477.454.358.040.206.064.427.500.389.661.640.321.194.300.307.110.230.904.655.559
14Pixtral-12B.688.501.530.349.187.029.079.027.447.495.320.465.599.352.307.304.229.266.246.065.119.519
15LLaVA-NeXT-Vicuna.662.343.290.100.028.003.015.003.457.454.364.198.510.250.217.250.701.125.187.000.000.500
16Phi-3.5-Vision.748.359.267.191.066.013.023.009.454.533.154.217.613.316.198.256.030.160.221.044.562.576
17Llama-3.2-Vision.545.482.518.434.234.031.104.054.317.483.286.520.724.286.260.317.061.125.250.722.000.500
18LLaVA-NeXT-Mistral.364.392.408.162.029.005.015.004.372.484.259.238.616.259.332.250.089.131.173.000.476.508
Mean.725.545.541.382.247.035.123.039.495.574.370.502.695.407.341.331.401.259.309.438.396.558
Floor.500.100.111.060.035.005.001.011.000.000.000.001.501.250.410.250.415.250.170.000.000.500

Table 3. Bio-VQA results for 18 models across 24 tasks. Colors mark the best, second best, second worst and worst model in each column, over the 18 models only. C1 is exact match at eight ranks. C2 is CAGS for relative salience (sal), morphological attributes (mrp), anatomical localization (loc) and naming (nam). C3 reports one metric per task: CAS for counting (cnt); balanced accuracy for relative counting (rcnt), relative distance (rdst) and relative position (rpos); the share of measurements within 20% of the true value for absolute distance (adst); mean IoU for localization (lcl) with unparsed replies scored as zero; agreement over gold labels for viewpoint (view); and exact match for OCR. C4 is discriminability (d′d') and accuracy under random pairing. Floor is the score an image-blind rule reaches, and the most frequent taxon at each rank for C1. CAGS and d′d' are chance-corrected, so floors are zero. ∗^*Domain specialist, taxonomy, with the full TreeOfLife-200M label set (tol). ∗∗^{**}Domain specialist, re-identification, on Grevy’s zebra only.

C1: VLMs fail at finer ranks

Exact match falls at every step down the hierarchy. The 18 models average .725 at phylum, .545 at class, .382 at family, .247 at genus and .123 at scientific name. This is a fall to one sixth over the same images. However, it is not ordered for every model: 9 models score higher at order than at class.

Coarse accuracy is inflated. Half our images are chordates, so an image-blind model answering Chordata to every image scores .500 at phylum. Three models score below that: InternVL3.5-30B at .486, Qwen2.5-VL-3B at .416 and LLaVA-NeXT-Mistral at .364. After subtracting the floor, models recover .450 of the headroom2 at phylum and .494 at class, although raw accuracy favors phylum by 1.3×. So the accuracy is inflated for phylum, partly because the answer distribution is narrower.

The collapse starts at finer ranks. The most common family covers 6% of the images, the most common genus 3.5% and the most common species 0.5%, so whatever a VLM scores here, it earns from the image. Even the best model reaches .612 at family and .269 at scientific name.

The specialist gap widens as the answer sharpens. The best model reaches .731 at class against BioCLIP-2’s .874, and .269 at scientific name against .544. The specialist advantage grows from 1.2× to 2.0×. BioCLIP-2 is trained on TreeOfLife-200M, the corpus our C1 images come from, and ranks a closed label set while VLMs answer open-ended, so we treat it as an optimistic upper bound than a baseline.

Scale helps mostly at coarse ranks. From 8B to 38B, InternVL3 gains 18.7 points at class and 6.5 at scientific name. InternVL3.5 gains 10.5 and 5.0 respectively. Both families gain 2× to 3× more at class than at scientific name3. Extra parameters can widen the coverage of categories these models partly knew, but they can not sharpen the finer distinctions they missed.

Models recognize what they cannot recall. On open-ended common name questions, models average .039. On the same images with four options, they average .645 (16.4×). The images and targets are identical, so the knowledge is present. VLMs can recognize the correct name among four options, but cannot retrieve it unprompted.

C2: VLMs see traits but cannot localize

Of the four tasks in C2, models average .574 on morphological attributes, .502 on naming, .495 on relative salience and .370 on anatomical localization. Naming a bird and finding its traits is easier than identifying a body part and its laterality. The two hardest sub-tasks are both from anatomical localization: part-set visibility (which parts of the bird are visible?) averages .341 and laterality (which of its eyes is visible?) .411. Why are these the hardest? A possible explanation is that these questions require more fine-grained reasoning. For example, from a coarser look, a model may answer the bird as Cardinal. And once it knows this, it may answer that the wing is red. Answering which of its two eyes is visible needs a finer look at the image.

Models confuse left and right. We take only the items where a model names the correct part, then ask which side it is on. 17 of the 18 models score at or below chance and 9 fall significantly below. Mirror confusion runs from 21.8% to 54.0% against a 20% baseline and rises with model strength (ρ=.75\rho=.75), so stronger models rule out the wrong part but then choose the mirror side. The one exception is Molmo 2-8B, the only model trained with explicit pointing supervision. So, VLMs find it difficult to localize a body part. Even when they can, they cannot tell which side it is.

The CAGS metric holds up. We sweep eleven kernel and hyperparameter settings and the model ranking barely moves (ρ≥.996\rho\geq.996 on the base grid, and ρ=.959\rho=.959 when anatomy similarity enters the visibility kernel). Three annotators rate 306 items and agree with CAGS at .908 of the rate they agree with each other. On the items exact match is zero, CAGS separates credited answers from rejected ones (AUC .676).

Models top on one task and tank on another. Molmo 2-8B leads marker localization at .743 and laterality at .746, the latter ahead of the next model by .220. But on naming it ranks 15th, scoring .330. Qwen2.5-VL-7B is the exact opposite, 2nd on naming at .713 and 12th on anatomical localization at .356. So, one finds the parts but cannot name the bird; the other names the bird but cannot find its parts.

C3: VLMs see objects but not their geometry

All eight tasks in C3 carry a floor, and models fall below the floor on two of them: localization and absolute distance.

Localization is the hardest and only Qwen3 does it well. Models average .401 mean IoU against a mean gold box at .415, and 12 of the 18 fall below it. Only Qwen3-VL-32B (.866), Qwen3-VL-8B (.862) and Qwen3-VL-30B (.861) find the animal reliably. Molmo 2-8B and Phi-3.5-Vision score lowest because most of their replies repeat the example box in the prompt.

Models recall a size instead of measuring one. Answering one fixed size per measurement type (elytral length and width) would score .410. Out of 18 models, 15 fall below this floor; only Qwen3-VL-32B (.622), Qwen3-VL-30B (.528) and InternVL3.5-38B (.503) beat it. The others answer in the right units but with far too little spread across. They recall a measurement instead of reading the bar.

Counting is close but rarely accurate. Models average .695 count agreement against a floor of .501, and InternVL3-38B leads at .799, but exact match never exceeds .242. VLMs find the right ballpark but not the exact count. The gap is specific to biological subjects. Qwen3-VL-32B counts 2 to 10 everyday objects exactly 94.9% of the time on CountBench [41, 43], but only 41.7% of single-image Bio-VQA items in the same range, 2.3× lower.

Relative tasks barely clear their floors. Relative counting averages .407 against a floor of .250 and relative distance .331 against .250. Relative position averages .259 against a guessing floor of .250 over its four options, so it stays at chance. VLMs see objects but cannot relate one to another.

OCR works as the negative control we designed. The seven spatial tasks correlate at mean ρ=+.703\rho= +.703 and none correlates with OCR. InternVL3-38B tops counting at .799 but drops to 14th on OCR at .105, so spatial composite measures spatial ability rather than general model competence.

C4: VLMs answer from resemblance

The specialist leads the field. We run MegaDescriptor on Grevy’s zebra and compare it against the VLMs. Under random pairing, it reaches d′d' .733 against a VLM mean of .391 (1.9×), and under easy pairing 2.055 against 1.346 (1.5×). A specialist re-identification model is about twice as good as the average VLM, although the two strongest VLMs exceed it on this dataset.

Under adversarial condition, models fall below chance. We pair different animals that sit closest in the DINOv2 embedding space, so the two match on pose, lighting and background. 17 of the 18 models then score negative d′d', from −.932 to −.091. Scoring below chance is not the error a guesser makes. It means a model reads a cue that we inverted. MegaDescriptor fails the same way at −.969. So, when the two animals that look most alike are the different ones, models keep calling them the same. Even specialists fail under adversarial conditions.

Difficulty ordering holds across models and species. 77 of 90 model-dataset cells (18×5) show the easy ≥ random ≥ hard ordering, against a 1/6 chance of monotone order. Response bias moves with it: criterion cc is negative for 17 of 18 models and grows more negative as pairs harden. Models answer yes more often as the task gets harder, not less.

Cross-component correlation

We correlate all 24 tasks over the 18 models to ask whether the four components measure four things. Table 4 reports the component-level correlations.

Spearman ρ\rhoC1C2C3C4
C1 (α\alpha .981)—.689.787.433†
C2 (α\alpha .901)—.927.514†
C3 (α\alpha .925)—.610
C4 (α\alpha .889)—

Table 4. Correlation between components over the 18 models scored on all four. α\alpha is each component’s internal consistency. † not significant after Holm correction. Confidence intervals are wide at n=18n = 18 (median width .66), enough to separate correlated from uncorrelated pairs but not to rank one pair above another.

Tasks agree more within a component than across one. Task pairs within a component correlate at mean ρ\rho .749, and pairs drawn from different components at .534. A label permutation puts p=.0005p = .0005 on that gap. Per component, the internal agreement is .841 for C1, .662 for C3, .645 for C2 and .617 for C4. Tasks within a component agree enough for the component score to summarize them, and differ enough to be worth reporting on their own.

C4 is the stand-alone component. It correlates with C1 at .433 and with C2 at .514, and neither survives multiple-comparison correction. Knowing how well a model names an organism tells us nothing about whether it can tell two individuals of that organism apart.

C2 and C3 overlap. The two correlate at .927. At the reliability we can measure, they are one construct. Both ask a model to find something fine-grained, so fine-grained attributes and spatial reasoning meet at the same skill. We keep them apart because they diagnose different failures, and we do not treat them as independent evidence.

A strong model is strong everywhere. A single factor explains 62.2% of the variance. InternVL3-38B, Qwen3-VL-30B and Qwen3-VL-32B place top-five in all components, so any of them is a safe pick whatever the task. Below them the ranking reorders: the median model moves 5.5 of 18 places between components, and Phi-3.5-Vision is at the 8th percentile on C1 and 64th on C4. A mid-tier model is good at one component and poor at another.

Discussion

Implications. Multiple choice says little about how a model answers on its own (.645 with four options, .039 without). Without a floor, we cannot tell whether a model looked at the image: on two tasks an image-blind rule beats most models. Where a specialist exists, it stays ahead of most VLMs.

Limitations. We evaluate open-weight VLMs only, so we cannot say if proprietary models like GPT-5 behave the same way. We run one decoding protocol, so we cannot say whether chain-of-thought or a reasoning model would change the results. Our images are from public datasets that a model may have seen in pretraining, including the BioCLIP-2 training corpus for C1. We report no human performance reference, and human raters validate C2 scoring only, with no human audit of C1, C3 and C4 ground truth. Our floors are rule-based, and a text-only VLM may set a higher one. Two models copy example answers on localization. Species-epithet answers are truncated, so we analyze species results with scientific name. Lastly, CAGS anatomy kernel over-credits laterality confusions (See Sec. 5.2).

Conclusion. In Bio-VQA, we ask 1.86M questions about 217K images, and after scoring the components against a floor or a specialist, we find that VLMs are good at coarse categories but poor at the finer ones, and scaling helps at coarse ranks. The gap to a specialist grows as the task gets more fine-grained. We release the benchmark4, the generators, the scorers5 and every per-item answer, along with Bio-VQA-Lite, a stratified sample of 500 questions per task (12K in total) for low-cost evaluation.

Acknowledgments

This research was supported by the Imageomics Institute, funded by the US National Science Foundation’s (NSF) Harnessing the Data Revolution (HDR) program under Award No. 2118240. And, Q. M. and Y. M. are supported by NSF under grant number NSF2406648.

Benchmark Statistics

C1: groups and question types

We draw 10 organismal groups from TreeOfLife-200M and take 10 000 images from each. Tab. 5 shows the filter that defines each group. Three groups need more than one filter value, because the GBIF Backbone holds no single accepted node that covers them.

Fish spans six classes. Fish is a paraphyletic concept and the backbone never treats it as one taxon. Actinopterygii is a valid class, but it covers the ray-finned fishes only and leaves out sharks, chimaeras, coelacanths and lungfish. We therefore assemble the group from six class values. Teleostei sits inside Actinopterygii in most treatments, and the backbone assigns both as class on different records, so we take both.

Reptiles carry no order. The backbone places Squamata, Testudines, Crocodylia and Sphenodontia at class rank, and puts Reptilia as a synonym of those four. Family

GroupFilter rankValue
AmphibiansclassAmphibia
AngiospermsclassMagnoliopsida, Liliopsida
ArachnidsclassArachnida
BirdsclassAves
BryophytesphylumBryophyta
FishclassActinopterygii, Teleostei, Elasmobranchii, Holocephali, Coelacanthi, Dipneusti
GymnospermsclassPinopsida
InsectsclassInsecta
MammalsclassMammalia
ReptilesclassSquamata, Testudines, Crocodylia, Sphenodontia

caption: 10 organismal groups in C1 and the backbone filter that defines each. Six of them do not appear in prior biological VQAs.

:::

Question typeScored rankExample answer
phylumphylumChordata
classclassAmphibia
orderorderAnura
familyfamilyHylidae
genusgenusDryophytes
species epithetspeciescinereus
scientific namescientific nameDryophytes cinereus
common namecommon namegreen treefrog
common name MCQcommon nameone of four candidates

caption: The nine C1 question types. We ask the common name twice on the same image, once open-ended and once as four-way multiple choice to measure the recognition–recall gap.

:::

sits directly under class, so the order column is empty for every reptile row. So we ask reptiles seven questions.

Angiosperms span two classes. The backbone splits the flowering plants into Magnoliopsida and Liliopsida, both at class rank under Tracheophyta. There is no node above them that holds the angiosperms alone, so we take both.

C1 spans 626 families, 2311 genera and 3403 species. This is 6×6 \times genus coverage and 4×4 \times species coverage of VLM4Bio. We ask nine question types per image. Tab. 6 shows each type, the rank it scores, and an example answer. Nine groups contribute nine questions and Reptiles contributes eight, which gives 890 000 questions in total.

C2: four categories, ten question families

We build C2 from CUB-200-2011, which records 312 binary attributes of 11 788 annotated birds. CUB-200-2011 stores every annotation as a part, a property and a value, such as has_wing_color::brown. We invert that triplet into a question and take the marked value as the answer. Based on the triplet, we divide our questions into four categories and 10 sub-categories. Tab. 7 shows the ten templates and Table 8 reports an example for each one.

FamilyQuestionOptionsAnswer
attr_bare_valueWhat is the bill color of this bird?white / yellow / black / pink / brownblack
attr_group_valueWhich of the following is the most prominently visible physical trait of the bird in this image?bill color: black / underparts color: white / primary color: green / breast color: red / eye color: whitebreast color: red
marker_localizationWhat is the body part of this bird marked with the yellow dot?beak / breast / left leg / left eye / left wingbeak
visibility_partsetWhich of the following parts of the bird are visible in this image?breast / beak / left wing / None / All threebeak
visibility_sideWhich of the eyes of the bird are visible in this image?Left / Right / Both / NoneBoth
category_mcqWhat type of bird is this?Waxwing / Nutcracker / Waterthrush / Grebe / GullGull
open_common_nameWhat is the common name of the bird?open-endedCalifornia Gull
open_scientific_nameWhat is the scientific name of the bird?open-endedLarus californicus
open_genusWhat is the genus of the bird?open-endedLarus
open_kind_ofWhat kind of gull is this?open-endedCalifornia

Table 8. One example per C2 family, all asked against the same photograph of a California gull. The four naming families ask the same bird at four levels of specificity, and only one of the four gives the model options.

The families do not share an answer space. color takes one of fifteen attested values, bill_length takes one of three, and visibility_side takes one of four. A raw graded score pays a model more for a lucky guess on a narrow item than on a wide one. This is what we correct per item in Sec. C.

A near miss implies something different in each family. Answering purple for blue is close on a perceptual scale. Answering nape for crown is close on an anatomical layout. Answering Left for Both is half of a set. No single kernel expresses all three, so we dispatch one kernel per family.

The four naming families allow us to ask naming questions at four levels of specificity on one image, from gull to Larus californicus. Only category_mcq supplies options, so the gap between it and open_common_name isolates recognition from recall without changing the image or the target.

We render a marker onto each of the 11 788 source images for anatomical marker localization questions, so C2 holds 23 576 images and 824 104 questions.

CategoryFamilyQuestionsFormat
Morphologicalattr_bare_value280 433MCQ
Rel. salienceattr_group_value321 073MCQ
Anatomicalmarker_localization11 788MCQ
Anatomicalvisibility_partset117 880MCQ
Anatomicalvisibility_side35 364MCQ
Namingcategory_mcq11 191MCQ
Namingopen_common_name11 788OE
Namingopen_scientific_name11 668OE
Namingopen_genus11 728OE
Namingopen_kind_of11 191OE
Total824 104

caption: C2 categories and question families. About 94% of items are multiple choice questions (MCQ) with two to five distractors. The other 6% are open-ended (OE) naming.

:::

C3: eight spatial tasks

We generate a C3 item only where an annotation carries a count, a measurement, a box or an angle. Table 9 reports the eight spatial tasks, and the dataset each one runs on. Table 10 gives an example per task.

#TaskItemsDataset
1Counting53 498all ten subsets
2Viewpoint13 000wildlife datasets
3Absolute distance14 108beetles
4Relative distance3976beetles
5Relative counting42 412all ten subsets
6Localization3988wildlife datasets
7Relative position1000beetles
8OCR522beetles
Total132 504

Table 9. C3 spatial tasks with the dataset each runs on. A task runs only where an existing annotation provides the ground truth, which is why five of them run on beetles.

Truth yyAnswer y^\hat{y}S(y^)S(\hat{y})EMsoft\mathrm{EM}_{\mathrm{soft}}
SquamataReptiliathe four reptile classes1
SquamataCrocodyliaCrocodylia0
Puma concolorFelis concolorPuma concolor1

Table 10.

Counting runs on three canvas tiers. A counting item is a canvas at a fixed pixel budget, so the encoder receives the same number of tokens whatever the tier. The Easy tier fills the canvas with one image. The Medium tier splits the canvas into two panels drawn from different datasets. The Hard tier splits it into two panels drawn from the same dataset. A target therefore occupies about half the area in Medium and Hard that it occupies in Easy. The three tiers isolate that area change from any change in content, and Sec. G reports what the split costs every model.

We macro-average counting over the ten subsets. The ten datasets do not contribute equal numbers of questions, they differ tenfold. GZCD contributes 2.9 times the median, because it produces three nested questions per image. Under micro-averaging, this one dataset supplies about 29% of the counting headline. Under macro-averaging it supplies 10%. So we report macro-average.

Absolute distance, relative distance and relative position run on the beetle trays, because that is the only dataset with recorded trait and scalebar measurements and where the specimens lie flat at one focal depth. Localization and viewpoint run on the C4 wildlife datasets, where one animal is in the foreground of the frame.

C4: five datasets, three conditions

We place two photographs side by side on one canvas and ask a single question.

Are these two photographs of the same individual animal? Answer Yes or No.

A model has to parse two subjects in one frame before it can compare them. This configuration is identical across models and conditions, so it does not confound the difficulty. Table 11 reports the five datasets with identity signal. Beluga is the one species with no unique pattern, and identify-

AnswerWhat it isEMEMsoft\mathrm{EM}_{\mathrm{soft}}sevwuplin
D. cinereusthe right species1101.001.00
Hyla cinereaits pre-2016 name0101.001.00
D. gratiosusbarking treefrog0010.830.75
A. americanusa toad, other family0030.500.35
Mus musculusa mouse, other class0050.170.05

Table 11.

ing a beluga requires reading accumulated scarring, which changes between sightings. This reflects on the results as reported in Sec. H, where beluga is the hardest dataset for every model we test. Each dataset contributes 1000 pairs per condition, balanced 500 same-individual against 500 different-individual, which gives 15 000 pairs in total. We draw the same-individual pairs first, because every dataset holds far fewer repeat sightings than distinct animals, so this is the scarce resource.

Source datasets

Every image and ground-truth annotation in Bio-VOQA comes from a public dataset. We generate a question only where an existing annotation supplies the answer, so no answer key depends on our own labeling. Tab. 12 gives the eleven sources and the component each serves.

What we release. Every image keeps the license of its source, so what we ship depends on where the ground truth sits. Most Bio-VOQA items ask about a source photograph. For these we ship an identifier and not the image. The identifier is a TreeOfLife-200M UUID for C1, a CUB filename for C2, and a catalog record id for C3 viewpoint, localization and OCR. This covers C1 in full and 9 of the 10 C2 families, which together hold most of the questions. We pin the UUID and the content hash of every C1 image, because our download runs concurrently under a budget per group and a fresh run would select a different 10,000 images per group. The rest of the items ask about an image we render, and for those the rendered image is the ground truth. C2 marker localization asks which part carries a dot we draw. C3 counting asks about a canvas we compose from one or two panels, and absolute and relative distance ask about lines we overlay on a beetle tray. Every C4 pair is two photographs we place side by side. In each case, the answer is correct only

TaskQuestionAnswer formAnswer
CountingHow many beetles are there in this image?integer11
Relative countingWhich of the 4 images contains the most number of beetles?one of four panelsImage B
Absolute distanceWhat is the measurement of the elytra width, marked with the red line, with respect to the scalebar, marked with the blue line, in cm?number in cm0.497
Relative distanceWhich beetle has the largest elytral length?one of four specimensBeetle D
LocalizationProvide the bounding box coordinate of the zebra in this image.four integers[854, 819, 3009, 2322]
Relative positionWhere is Beetle (blue) located with respect to Beetle (red)?one of eight directionsTop-right
ViewpointWhat is the viewpoint of the hyena?one of ten orientationsleft
OCR (control)What is the barcode text?alphanumeric stringA00000012435

caption: One example per C3 task. Five of the eight take an open-ended answer, so a model cannot recover the answer form from a list of options. The answer form column is what the parser expects, and Sec. K gives the full template for each.

:::

datasetSpeciesIdentity signal
WhaleSharkIDwhale sharkspot pattern
BelugaIDbeluga whalescarring, dorsal ridge
HyenaIDspotted hyenaspot pattern
LeopardIDleopardrosette pattern
GZCDGrevy’s zebrastripe pattern

caption: The five C4 datasets and the marking that identifies an individual in each. Four carry a pattern that is unique to the animal and stable across sightings. Beluga is the exception.

:::

DatasetUsed byContent
TreeOfLife-200MC1100K subset, ten groups
CUB-200-2011C211 788 annotated birds
WAIDC3aerial wildlife
Boxes on BeesC3dense hive frames
2018-NEON-beetlesC3trays with scalebars
DePondFiC3pond fish
GZCDC3, C4Grevy’s zebra
WhaleSharkIDC3, C4whale shark
HyenaIDC3, C4spotted hyena
LeopardIDC3, C4leopard
BelugaIDC4beluga whale

caption: The eleven public datasets behind Bio-VQA. Four of them serve two components.

:::

for the image we produced, so we redistribute those images under the terms of their upstream sources, with per-source attribution in the release.

Models and Protocol

The 18 open-weight models

Table 13 lists the 18 models we evaluate, with the checkpoint, the language backbone and the vision path of each.

ModelCheckpointParamsLLM backboneVision path
Qwen2.5-VL-3BQwen/Qwen2.5-VL-3B-Instruct3.75BQwen2.5-3BNative-resolution ViT (0.67B), window attention
Qwen2.5-VL-7BQwen/Qwen2.5-VL-7B-Instruct8.29BQwen2.5-7BNative-resolution ViT (0.67B), window attention
Qwen3-VL-8BQwen/Qwen3-VL-8B-Instruct8.8BQwen3-8BQwen3-VL ViT, DeepStack fusion
Qwen3-VL-30B-A3BQwen/Qwen3-VL-30B-A3B-Instruct30.5B / 3B activeQwen3-30B-A3BQwen3-VL ViT, DeepStack fusion
Qwen3-VL-32BQwen/Qwen3-VL-32B-Instruct32BQwen3-32BQwen3-VL ViT, DeepStack fusion
InternVL3-8BOpenGVLab/InternVL3-8B-Instruct8BQwen2.5-7BInternViT-300M-448px-V2.5
InternVL3-38BOpenGVLab/InternVL3-38B-Instruct38BQwen2.5-32BInternViT-6B-448px-V2.5
InternVL3.5-8BOpenGVLab/InternVL3.5-8B-Instruct8BQwen3-8BInternViT-300M-448px-V2.5
InternVL3.5-30B-A3BOpenGVLab/InternVL3.5-30B-A3B-Instruct31B / 3B activeQwen3-30B-A3BInternViT-300M-448px-V2.5
InternVL3.5-38BOpenGVLab/InternVL3.5-38B-Instruct38.4BQwen3-32BInternViT-6B-448px-V2.5
Gemma-3-12Bgoogle/gemma-3-12b-it12BGemma 3 12BSigLIP-So400m/14-896, pan-and-scan
Gemma-4-12Bgoogle/gemma-4-12B-it12BGemma 4 12BEncoder-free. A 35M embedder projects raw 48×48×3 patches straight into the LLM embedding space
Molmo2-8Ballenai/Molmo2-8B8.66BQwen3-8BSigLIP 2
Pixtral-12Bmistral-community/pixtral-12b12.4BMistral NeMo 12BPixtral-ViT (0.4B), trained from scratch, variable resolution
Phi-3.5-Visionmicrosoft/Phi-3.5-vision-instruct4.2BPhi-3.5-mini (3.8B)CLIP ViT-L/14-336, multi-crop
Llama-3.2-Visionmeta-llama/Llama-3.2-11B-Vision-Instruct10.6BLlama 3.1 8B, frozenViT-H/14 (0.63B) with a cross-attention adapter
LLaVA-NeXT-Vicunallava-hf/llava-v1.6-vicuna-7b-hf7.06BVicuna-7B-v1.5CLIP ViT-L/14-336, AnyRes tiling
LLaVA-NeXT-Mistralllava-hf/llava-v1.6-mistral-7b-hf7.57BMistral-7B-Instruct-v0.2CLIP ViT-L/14-336, AnyRes tiling

Table 13. The 18 open-weight models, from 3.75B to 38.4B parameters. Two are mixture-of-expert variants that activate 3B parameters per token.

Two groupings in Table 13 give us controlled comparisons: Three models share one backbone. Qwen3-VL-8B, InternVL3.5-8B and Molmo2-8B all run Qwen3-8B. Their vision paths differ. Qwen3-VL uses a DeepStack ViT, InternVL3.5 uses InternViT-300M, and Molmo2 uses SigLIP 2. Molmo2-8B leads the field on marker localization and on laterality in Sec. F. The language model is the same in all three, so that lead comes from the vision path and not from the backbone.

Two pairs share a vision encoder. InternVL3-38B and InternVL3.5-38B both use InternViT-6B as vision encoder. InternVL3-38B runs Qwen2.5-32B as LLM backbone, whereas InternVL3.5-38B runs Qwen3-32B. InternVL3-8B and InternVL3.5-8B do the same with InternViT-300M.

One model has no vision encoder. Gemma-4-12B projects raw pixel patches into the language embedding space with one matrix multiply. The embedder holds 35M parameters. The model leads relative salience in Sec. F and relative distance in Sec. G, and is a case against assuming that coarse visual judgement needs a large pretrained encoder.

Protocol and Environment

Decoding protocol. Every model runs under one protocol on every component. We use greedy decoding with sampling disabled. We use a fixed system prompt per question family. We use no chain-of-thought and no few-shot examples. We allow one attempt per item. One parser reads all outputs. We choose this protocol so that the benchmark measures what a model produces under a single, cheap, reproducible setting. A model that improves with chain-of-thought or with a reasoning mode may well score higher, and we say so in Sec. M. What we measure is the zero-shot single-pass answer.

Environments. All 18 models run in one shared Python 3.12 environment, except Molmo2-8B. AIZ ships Molmo2 as custom code pinned to transformers 4.57.1.

Response Parsing

We report two rates. The parse rate is the percentage of replies from which the parser recovers an answer. The letter-text mismatch rate is the percentage of replies where a model yields a letter and an option text that disagree, such as (B) brown when B is black in the original option set.

Mismatch only exists where a question offers lettered options. Most of Bio-VQA does not. C1 asks eight of its nine types open-ended, so only the common-name multiple choice can mismatch. C3 asks five of its eight tasks open-ended, so only relative counting, relative distance and relative position can mismatch. C4 takes a yes or a no and carries no letters at all, so mismatch is undefined for the whole component. C2 is the one component where most items are multiple-choice and lettered. Table 14 reports both rates along with answer form.

Component and taskAnswer formParseMismatchParsing Criteria
C1, eight rank typesopen text1.000—We match answers with ground truth after normalization (The phylum is Chordata. normalizes to chordata)
C1, common name MCQbare options1.000—We match the answer against the four option strings
C2, six MCQ subtaskslettered options.960.0117A mismatch in letter and text. Original options: (A) Red, (B) Blue. Answer: (B) Red.
C2, four open subtasksopen text1.000—We accept any string and score it after normalization
C3, countingopen integer.987—We reject an answer that holds no integer, such as There are many beetles
C3, viewpointopen text1.000—10 candidates given in prompt. We reject an answer that holds none.
C3, absolute distanceopen number.970—We reject an answer that holds no number in centimeters
C3, relative distancenon-lettered options.642—We reject an answer that names none of the four specimens
C3, relative countingnon-lettered options1.000—4 candidates given in prompt. We reject an answer that names none (Image A/B/C/D)
C3, localizationfour integers.523—We reject an answer that holds no box, such as The zebra is in the center
C3, relative positionnon-lettered options1.000—8 candidates given in prompt. We reject an answer that holds none.
C3, OCRcode string.996—We reject an answer that holds no code-like string
C4yes or no1.000—We reject an answer that does not contain a clear Yes or No answer.6

Table 14. Parse quality: We report the model with lowest parsing score in each component and task. Mismatch needs a letter and an option text that can disagree, so only C2 reports it. Parsing Criteria column shows what the parser accepts or rejects for that answer form. 6 In C4, 269 903 answers are Yes or No. The other 97 contains Yes or No but in a sentence (e.g. Yes, they are the same).

C1 parses at 1.000 for a structural reason. An open-ended answer is a string, and any string compares against the gold string, so nothing fails to parse. On C4 the parser reads a bare Yes or No in 269 903 of the 270 000 answers. The other 97 need one loose rule, which accepts yes or no appearing as a word inside a longer reply.

Specialist baselines

We bound two components from above with specialists. BioCLIP-2 bounds C1 and MegaDescriptor bounds C4.

BioCLIP-2 for C1. BioCLIP-2 is a contrastive vision model. It trains on the TreeOfLife-200M dataset with hierarchical taxonomic supervision. We choose this as the specialist for two reasons. First, it is in-domain, so its accuracy approximates the best a current model reaches on this label space. Second, its supervision is the Linnaean hierarchy, so its predictions carry a full lineage and we score it at every rank with the machinery we use for the VLMs.

We get answers from one species prediction. BioCLIP-2 is a classifier and cannot read a prompt. We take top-1 species per image and read the eight coarser ranks off that species lineage. For multiple choice, we restrict a custom-label classifier to the four options and take the arg ⁣- ⁣max⁡\operatorname{arg\!-\!max}. This configuration implies a coarse rank is only correct when the predicted species shares true lineage at that rank.

We test two configurations. We evaluate the specialist under two candidate sets. Table 15 lists the three. We have not evaluated the third one, but list as a potential ceiling.

SpaceCandidatesSizeRole
kingdomspecies in the named kingdom10410^{4}–10510^{5}fair
tolall of TreeOfLife-200M∼9×105\sim9 \times10^{5}ablation
groupthe group’s gold species10210^{2}ceiling

Table 15. The three candidate sets for BioCLIP-2. We report tol in the main paper, which gives BioCLIP-2 no kingdom cue; kingdom differs from it by +0.060 at phylum and by at most 0.005 at every finer rank.

Kingdom–TOL gap. In BioCLIP-2/kingdom, we restrict the candidate set to only the kingdom of the target taxon. In BioCLIP-2/tol, there is no restriction. So without the kingdom clue, a plant image can match an animal whose name aligns in embedding space. A Pinus strobus image classifies as the white pine barkminer moth, an insect named after the tree it feeds on. We test both configurations and find the gap is very little. BioCLIP-2/kingdom and BioCLIP-2/tol differ by +0.060 at phylum, which is 0.985 against 0.925, and by almost nothing at every finer rank. So the clue prevents gross cross-kingdom errors and gains nothing below phylum.

The specialist is far from perfect at species. BioCLIP-2 reaches .545 at species and not 1.0. We identify four reasons for this. Huge candidate space. Species prediction is a top-1 decision over 10510^{5} labels under kingdom restriction and 10610^{6} over the full tree. The correct species competes against many near-identical neighbors, and we take top-1. So ranking it second scores zero. Long-tailed dataset. The training distribution is severely imbalanced. A species seen a handful of times cannot acquire </@body a sharp decision boundary, even though its images are nominally in the training set. Cryptic species. Cryptic species can look identical in one photograph. The discriminating signal is not in the pixels. Contrastive objective. CLIP-style training aligns image and text embeddings. It does not optimize per-image label recall. MegaDescriptor for C4. MegaDescriptor-L-384 is a purpose-built wildlife re-identification embedding. We compare two images by cosine similarity and threshold it. We fit the threshold on the random condition and apply it unchanged to easy and hard, so those two rows are out of fit. AUC needs no threshold and converts to the units of the VLM table through d′=2 z(AUC)d' = \sqrt{2}\,z(\mathrm{AUC}). We run MegaDescriptor on Grevy’s zebra only, so we compare it against the 18 models on that dataset alone.

Metric Definitions

We give every equation we use, with a worked example for each. The main paper carries four of these. We report the rest here.

C1: Exact match

For a normalized prediction y^\hat{y} and ground truth yy,

EM=1[y^=y](5)\mathrm{EM} = \mathbb{1}[\hat{y}=y] \tag*{(5)}

We normalize both ground truth and predicted answers before we compare them. The normalizer case-folds, strips diacritics, removes punctuation, and removes leading boilerplate and rank prefixes. Under this map ‘The phylum is Chordata.’ and ‘chordata’ compare equal.

C1: Soft exact match and the synonym relation

Taxonomy changes. A species gets a new taxonomic name for the same animal. The green treefrog was Hyla cinerea until 2016 and has been re-taxonomized to Dryophytes cinereus. Both names appear in the literature, and a model trained on text written before the change learns the old one. Strict exact match calls the old name wrong, which is too strict. A model named the right animal but used a name the current backbone no longer contains. So we add a second column that credits it.

Let S(y^)S(\hat{y}) be the set of accepted taxa that y^\hat{y} resolves to in the GBIF Backbone. Then

EMsoft=1[y^=y∨y∈S(y^)](6)\mathrm{EM}_{\mathrm{soft}} = \mathbb{1}[\hat{y}=y \lor y \in S(\hat{y})] \tag*{(6)}

The gap EMsoft−EM\mathrm{EM}_{\mathrm{soft}}-\mathrm{EM} measures how much of a model’s apparent error is stale nomenclature and not absent knowledge. We report it Sec. D. The gap widens exactly in the groups whose higher taxonomy was revised. The synonym mapping has two properties to address:

It returns a set. One taxon name can split into several modern taxa, which taxonomists call a pro-parte synonym. Reptilia is the clearest case,

S(reptilia)={squamata, testudines, crocodylia, sphenodontia}.(7)S(\mathit{reptilia}) = \{\mathit{squamata},\ \mathit{testudines},\ \mathit{crocodylia},\ \mathit{sphenodontia}\}. \tag*{(7)}

Any of these four (lizard, turtle, crocodile, tuatara) answered Reptilia are therefore credited. A map that stored one target per name would keep one of the four and reject the other three. In the Backbone snapshot, 75 315 names carry two or more accepted targets.

It runs one way. We look at the prediction and check whether it resolves onto the ground truth. We do not credit the prediction just to be in the same synonym cluster as the ground truth (e.g. Crocodylia and Squamata are in the same cluster). We do this to prevent over-crediting the models. As we see in Eq. (7), Reptilia resolves onto 4 classes. A cluster rule would link Squamata and Crocodylia to each other through Reptilia, and it would credit a model that answers Crocodylia for a lizard. The one-way rule credits Reptilia for a Squamata. It rejects that, because Crocodylia is an accepted name and resolves onto itself.

C1: Graded tree distance

Exact match and soft match are binary. They cannot tell a near miss from a distant one. So we place the prediction and the truth in the taxonomic tree and measure how far apart they sit. Let d(⋅)d(\cdot) be depth from the root and let ℓ=LCA⁡(y^,y)\ell= \operatorname{LCA}(\hat{y},y) be the lowest common ancestor.

Mistake severity counts how many ranks we climb from the truth before we meet the prediction,

sev=E ⁣[d(y)−d(ℓ)  |  y^≠y](8)\mathrm{sev} = \mathbb{E}\!\left[d(y) - d(\ell) \;\middle|\; \hat{y} \ne y\right] \tag*{(8)}

A severity of 1 means the misses stay inside the correct parent. Higher values mean a model misses the broader clade too. We keep severity because it answers a question accuracy cannot. When a model is wrong, how wrong is it. A model that names the wrong frog family scores 1. A model that names a mouse scores 3.

Wu–Palmer similarity normalizes the depth of the shared ancestor:

wup⁡(y^,y)=2d(ℓ)d(y^)+d(y)(9)\operatorname{wup}(\hat{y},y) = \frac{2d(\ell)}{d(\hat{y}) + d(y)} \tag*{(9)}

Wu–Palmer asks how deep the agreement runs. Two answers that meet at the genus score higher than two that meet at the class. It turns severity into a bounded similarity, so we can average it across ranks that sit at different depths.

Lin similarity weights the agreement by how rare the shared ancestor is, so agreeing on an obscure clade counts for more than agreeing on a common one.

lin⁡(y^,y)=2IC⁡(ℓ)IC⁡(y^)+IC⁡(y),IC⁡(c)=−log⁡∣subtree⁡(c)∣N(10)\operatorname{lin}(\hat{y},y) = \frac{2\operatorname{IC}(\ell)}{\operatorname{IC}(\hat{y}) + \operatorname{IC}(y)}, \quad\operatorname{IC}(c) = -\log\frac{|\operatorname{subtree}(c)|}{N} \tag*{(10)}

Here NN is the total node count. Wu–Palmer treats every rank as equally informative. Lin does not. Half our images are chordates, so agreeing on Chordata is easier than agreeing on a small genus. Lin computes that difference.

Worked example. We show a green treefrog and we ask for its species. The answer is Dryophytes cinereus. Suppose five models answer. We compute Lin-similarity with round subtree counts. We take N=900 000N = 900\,000 nodes, with Chordata covering half the tree, Anura covering 7500 nodes, Dryophytes covering 30, and each species covering one. From the scores, we see that:

  • EM credits one answer out of five. It calls the pre-2016 name wrong, and it gives the barking treefrog, the toad and the mouse the same zero.

  • EMsoft\mathrm{EM}_{\mathrm{soft}} fixes the first problem. Hyla cinerea resolves onto Dryophytes cinereus in the backbone, so the model gets its point. It is the same animal under an older name. EMsoft\mathrm{EM}_{\mathrm{soft}} leaves the other three at zero, because they are different species.

  • Severity fixes the second problem. The three wrong answers are not equally wrong. One is a treefrog in the same genus. One is a toad in another family. One is a mouse. Severity scores 1, 3, 5.

  • Wu–Palmer turns that into a similarity we can average, and it keeps the ordering.

  • Lin rescores them based on rarity of the taxon. Wu–Palmer awards the mouse 0.17 for sharing Chordata. But this it is over-credited, because half our images are chordates. Lin awards it 0.05. Lin also pulls the toad down from 0.50 to 0.35, because Anura is a large clade. It leaves the barking treefrog near the top, because Dryophytes is a smaller clade.

So the columns answer four different questions: 1. Is the answer right? 2. Is it right under an older name? 3. How far up the tree does it miss? 4. How much does the part it got right tell us?

C2: Chance-adjusted graded score

A raw graded score fails C2 in two ways. First, answer spaces have structure. On a brown wing, red is a better answer than green. A binary score calls both wrong. Second, answer spaces differ in size. Bill length offers 3 values and color offers 15. A image-blind model that guesses on a 3-way item scores better than a model that guesses on a 15-way item. CAGS fixes both. We score similarity with a family kernel, then we subtract what an image-blind guess earns on that same item. Let s(y^,y)∈[0,1]s(\hat{y},y) \in[0,1] be the kernel and let bb be the mean similarity of the item’s own distractors to the ground truth,

b=1∣D∣∑d∈Ds(d,y)(11)b = \frac{1}{|D|}\sum_{d \in D} s(d,y) \tag*{(11)}

DD is the set of distractors for a multiple-choice item. For an open-ended item we draw a fixed-seed sample of 40 other ground truth answers from that family. Then

CAGS⁡i=si−bi1−bi,CAGS⁡‾=1N∑i=1NCAGS⁡i(12)\operatorname{CAGS}_{i} = \frac{s_i - b_i}{1-b_i}, \qquad\overline{\operatorname{CAGS}} = \frac{1}{N}\sum_{i=1}^{N}\operatorname{CAGS}_{i} \tag*{(12)}

A CAGS of 1 is a perfect answer. A CAGS of 0 is exactly the chance level for that item. A negative CAGS means the answer sits farther from the truth than a random chance.

Worked example. We ask for a wing color. The truth is blue and the distractors are purple, green and yellow. The color kernel in Eq. (13) gives s(purple,blue)=0.498s(\mathrm{purple},\mathrm{blue}) = 0.498, s(green,blue)=0.134s(\mathrm{green},\mathrm{blue}) = 0.134 and s(yellow,blue)=0.053s(\mathrm{yellow},\mathrm{blue}) = 0.053, so b=0.228b = 0.228.

Based on Eq. (12) and the bb value, we calculate the CAGS scores and report them in Tab. 16. We keep the negative numbers and do not clip them at zero. Clipping would put green and yellow at

AnswerssCAGSReading
blue1.0001.000correct
purple0.4980.350better than a guess
green0.134−0.122worse than a guess
yellow0.053−0.227much worse than a guess

the same score, and the severity metric in Eq. (22) depends on the difference in mistakes. A model that scores negative is confidently wrong, not uncertain. We want to test and report it.

Also, the difficulty of the question lies in the distractor set. When every distractor is close to the truth, bb is high and a decent answer gets partial score in CAGS. When the distractors are far away, bb is low and only a good answer gets credited. Exact match cannot do this. We also report the mean raw score GSraw. The gap GSraw − CAGS grows when a family of question is easy to look good on (See Sec. C.5).

C2: The six similarity kernels

Every question family has its own idea of a near miss, so each gets its own kernel.

Color. We anchor each of the 15 CUB color words to an sRGB triple. We push it through inverse gamma, linear RGB, XYZ under D65, and CIE-Lab. Then we compare two colors with CIEDE2000 ΔE00\Delta E_{00}, which is a perceptual distance and not a distance between the words,

color⁡(c1,c2)={1c1=c20either is iridescentexp⁡(−ΔE00/τ)τ=30 defaultmax⁡(0,1−ΔE00/ΔEmax⁡)linear(13)\operatorname{color}(c_1,c_2) = \begin{cases} 1 & c_1 = c_2 \\ 0 & \text{either is iridescent} \\ \exp(-\Delta E_{00}/\tau) & \tau= 30\ \text{default} \\ \max(0,1-\Delta E_{00}/\Delta E_{\max}) & \text{linear} \end{cases} \tag*{(13)}

Iridescent has no fixed hue, so we score it by exact match only.

Pairexponentiallinear
blue, purple0.4980.791
blue, green0.1340.396
blue, yellow0.0530.120
brown, rufous0.7600.918
yellow, orange0.5070.796
black, white0.0360.000

Table 5.

Ordinal. Bill length and size run on an ordered scale, so we use normalized rank distance,

ord⁡(v1,v2)=1−∣rank⁡(v1)−rank⁡(v2)∣rank⁡max⁡(14)\operatorname{ord}(v_1,v_2)=1-\frac{\lvert\operatorname{rank}(v_1)-\operatorname{rank}(v_2)\rvert}{\operatorname{rank}_{\max}} \tag*{(14)}

Bill length shorter against about the same scores 0.50, and shorter against longer scores 0. Size small against medium scores 0.75, so an off-by-one size error keeps three quarters of its credit.

Pattern. Patterns have no order, so rank distance does not apply. We set the values by hand from one fact. A bird that is spotted, striped or multicolored has a patterned surface. A bird that is solid has none. So confusing two patterns is a smaller error than calling a patterned bird solid, so spotted for striped keeps half its credit

and solid for striped keeps none.

pat⁡(v1,v2)={1.0v1=v20.5v1,v2∈{spotted, striped, multicolored}0.0otherwise(15)\operatorname{pat}(v_1,v_2)= \begin{cases} 1.0 & v_1=v_2 \\ 0.5 & v_1,v_2\in\{\text{spotted, striped, multicolored}\} \\ 0.0 & \text{otherwise} \end{cases} \tag*{(15)}

Anatomy. Body parts sit at coordinates in a three-dimensional layout. The bird faces right, yy runs down, and zz is a small lateral offset that separates left from right. Part similarity is normalized proximity in that layout,

part⁡(p1,p2)=max⁡(0,1−∥p1−p2∥2dmax⁡)(16)\operatorname{part}(p_1,p_2)=\max\left(0,1-\frac{\lVert p_1-p_2\rVert_2}{d_{\max}}\right) \tag*{(16)}

where dmax⁡d_{\max} is the layout diameter. We build the layout from the data. For each part we take the median bounding-box-normalized coordinate over every image where CUB records that part as visible.

Pairpart similarity
nape, crown0.922
beak, forehead0.857
left wing, right wing0.883
crown, belly0.430
beak, tail0.000

Table 6.

We use this kernel on anatomical marker localization, where it credits crown for nape at 0.92 and tail for nape at 0. Exact match gives both a flat zero, which erases the difference between a model that nearly found the marker and one that was lost. The left wing and right wing pair scores 0.883 and not 1.000 because of the zz offset. This is a limitation of the CAGS metrics which we report in Sec. F.

Paired attribute. A relative salience option names a region and a value together, so a near miss can be in the region, in the value, or in both. We score the two halves separately and combine them,

pair⁡=α g(r1,r2)+(1−α) v(v1,v2)α=0.5(17)\operatorname{pair}=\alpha\,g(r_1,r_2)+(1-\alpha)\,v(v_1,v_2)\qquad\alpha=0.5 \tag*{(17)}

where gg blends the anatomy similarity of the two regions with a property-match indicator, and vv dispatches to the color, ordinal or pattern kernel. Wing color: blue scores 0.749 against wing color: purple, 0.952 against breast color: blue, and 0.395 against tail color: yellow. Getting the region right and the color close beats getting the color right and the region wrong.

Set overlap. A visibility answer is a set, so we use a soft Jaccard,

sJac⁡(A,B)=∑a∈Amax⁡b∈Bκ(a,b)∣A∣+∣B∣−∑a∈Amax⁡b∈Bκ(a,b)(18)\operatorname{sJac}(A,B)=\frac{\sum_{a\in A}\max_{b\in B}\kappa(a,b)}{\lvert A\rvert+\lvert B\rvert-\sum_{a\in A}\max_{b\in B}\kappa(a,b)} \tag*{(18)}

with element kernel κ\kappa equal to exact match by default. Both against Left scores 0.50, because the model named one of the two. Both against None scores 0.00, and Left against Right scores 0.00.

Naming. In C2, three question families ask about the bird’s common English name or what type of bird it is, and two about the taxonomy at genus and species ranks (See Tab. 7). Although the task seems similar to C1, we do not reuse the C1 tree metrics. The metrics in C1 use a taxonomic lineage that comes from the TreeofLife-200M dataset. Whereas ground truth answers in C2 come from CUB-200-2011 dataset and has no information on lineage, so there is no tree to evaluate the Wu–Palmer and Lin. A label in C2 is a bird name, not a node. So we score the names directly.

The CUB name mapping. CUB-200-2011 gives a class a common English name. There is no scientific name. So we build a map from each of the 200 CUB classes to its accepted scientific name, we verify every entry against both the iNaturalist taxonomy and Wikidata. We release the map with the benchmark.

Metrics. Let T(⋅)T(\cdot) be the set of lowercased tokens in a name. Token F1F_1 is the harmonic mean of token precision and recall:

F1(p,g)=2 ∣T(p)∩T(g)∣∣T(p)∣+∣T(g)∣(19)F_1(p,g) = \frac{2\,\lvert T(p) \cap T(g)\rvert}{\lvert T(p)\rvert+ \lvert T(g)\rvert} \tag*{(19)}

Song sparrow against fox sparrow shares one token out of two on each side, so F1=2/4=0.5F_1 = 2/4 = 0.5. We then score the three name forms. Here head(⋅)(\cdot) is the last token of a common name, which is the noun the name ends on,

common(p,g)=0.6 F1(p,g)+0.4 1 ⁣[head(p)=head(g)]sci(p,g)=0.5 agen+0.5 aepigenus(p,g)=agen(20)\begin{aligned} \mathrm{common}(p,g) &= 0.6\,F_1(p,g) + 0.4\,\mathbf{1}\!\left[\mathrm{head}(p) = \mathrm{head}(g)\right] \\ \mathrm{sci}(p,g) &= 0.5\,a_{\mathrm{gen}} + 0.5\,a_{\mathrm{epi}} \\ \mathrm{genus}(p,g) &= a_{\mathrm{gen}} \tag*{(20)} \end{aligned}

A component that agrees scores 1. A component that disagrees falls back to a down-weighted string similarity, so a wrong genus that looks like the right one earns a partial score:

ax={1the two agree on x0.3 ratio(xp,xg)otherwise(21)a_x = \begin{cases} 1 & \text{the two agree on } x \\ 0.3\,\mathrm{ratio}(x_p,x_g) & \text{otherwise} \end{cases} \tag*{(21)}

where ratio is the sequence-matching ratio between the two strings. We lift the common-name score to at least 0.75 when the two names share a genus, because two birds in one genus are close whatever their common names look like.

The last two rows in Table 17 show the fallback effect. Melospiza georgiana keeps 0.5 from the matching genus and adds 0.5×0.3×0.5=0.0750.5 \times0.3 \times0.5 = 0.075 from the epithet, which gives 0.575. Corvus brachyrhynchos agrees on nothing, so both halves fall back and it lands near zero.

PredictionGround truthScoreReason
song sparrowfox sparrow0.700F1=0.5F_1 = 0.5, same head
american crowfox sparrow0.000no shared token, other head
M. georgianaM. melodia0.575genus agrees, epithet fallback
C. brachyrhynchosM. melodia0.054both components fallback

Table 17.

Mistake severity. We average over wrong items only,

Severity=1Nwrong∑i:EMi=0(1−CASi).(22)\mathrm{Severity} = \frac{1}{N_{\mathrm{wrong}}}\sum_{i:\mathrm{EM}_i=0}(1-\mathrm{CAS}_i). \tag*{(22)}

A severity at or below 1 means the average miss better than a random guess. A severity above 1 means the average miss lands farther from the truth than a random guess. We report the severity numbers in Sec. F.

C3: Spatial reasoning metrics

Bounded count agreement. A model predicting 14 beetles when the truth is 15 shows better spatial reasoning than one predicting 10. We address this using relative error:

CAS(n^,n)=max⁡(0,1−∣n^−n∣n).(23)\mathrm{CAS}(\hat{n},n)=\max\left(0,1-\frac{\lvert\hat{n}-n\rvert}{n}\right). \tag*{(23)}

We then macro-average the scores across datasets. We do this because dataset sizes vary widely. For instance, the GZCD dataset produces three questions per image. If we micro-average, GZCD dominates 29% of the final score. Macro-averaging reduces this to a fair 10%.

truepredCASEM
10101.0001
10120.8000
10150.5000
10200.0000
10900.0000
100950.9500

The image-blind floor and lift. Every C3 task has a performance floor. We define this floor using simple rules that do not look at the image, like guessing the most common count or drawing a box over the whole image. Let G0\mathcal{G}_0 be this set of image-blind rules. We define the floor ϕ\phi and the resulting lift as:

ϕ=max⁡g0∈G0E[S(g0,y)],lift=S−ϕ1−ϕ.(24)\phi= \max_{g_0\in\mathcal{G}_0}\mathbb{E}[S(g_0,y)], \qquad\mathrm{lift}=\frac{S-\phi}{1-\phi}. \tag*{(24)}

A lift of 0 means the model performs exactly like a blind guess. A negative lift means it performs worse. We calculate this baseline strictly on the rows the model successfully passes. If a model fails to format answers for some rows, we drop those rows and compute a fair baseline on the remainder.

Viewpoint agreement. There are 10 possible viewpoint orientation. In C3, the right viewpoint dominates. Because of this skew, raw accuracy is misleading. To address this, we assign partial credit for near misses and average the scores across all ground truth labels. We write v(y,y^)v(y,\hat{y}) for the graded credit of a predicted viewpoint against a gold one, and L′\mathcal{L}' for the gold labels present:

VAS=1n∑iv(yi,y^i),\mathrm{VAS}=\frac{1}{n}\sum_i v(y_i,\hat{y}_i),
macroVAS=1∣L′∣∑ℓ∈L′1nℓ∑i:yi=ℓv(yi,y^i).(25)\mathrm{macroVAS}=\frac{1}{\lvert\mathcal{L}'\rvert}\sum_{\ell\in\mathcal{L}'}\frac{1}{n_\ell}\sum_{i:y_i=\ell}v(y_i,\hat{y}_i). \tag*{(25)}

This macro-averaging gives rare and common viewpoints equal weight. Next, we correct this score for chance against a model that always guesses the same answer, yielding κ\kappaVAS. This is our headline metric. For example, a rule that always guesses right gets a 52% exact match accuracy, but a 0 score in κ\kappaVAS. Finally, we also measure angular error. Both metrics rank the models identically, confirming our scoring method works.

ControlStrategy it catchesScore under it
per-type constant floorone value for length, another for widthacc⁡@20=0.410\operatorname{acc}@20 = 0.410
ρwithin\rho_{\mathrm{within}}any predictor constant within a typeρwithin=0\rho_{\mathrm{within}} = 0
compression κ\kappaone remembered size for both typesκ≈1\kappa\approx1

Table 7.

Absolute distance. Models can cheat on distance tasks by memorizing priors. We use three checks to resolve this. We use ρwithin\rho_{\mathrm{within}} (Spearman correlation) to rank predictions within each measurement type. If a model just guesses a constant average size for every beetle, it gets a correlation of exactly zero. We use κ\kappa compression to catch models that use one single size for everything. Finally, we report acc⁡@20\operatorname{acc}@20, which is the percentage of answers within 20% of the true length. Together, these checks ensure a model only gets a high score if it actually measures the image.

Localization. We score localization using mean Intersection over Union (IoU) against the ground truth box. We report the share of predictions passing various IoU thresholds (acc⁡@.25\operatorname{acc}@.25, acc⁡@.50\operatorname{acc}@.50, acc⁡@.75\operatorname{acc}@.75, and acc⁡@.90\operatorname{acc}@.90). Our main metric is meanIoU⁡all\operatorname{meanIoU}_{\mathrm{all}}. We calculate this by multiplying the mean IoU by the model’s parse rate. We penalize unparsed answers with a zero score instead of ignoring them. Models output bounding boxes in different formats (like absolute pixels or normalized coordinates). We test four common formats for each model and pick the one that gives the highest score. Getting this format wrong drops a model’s score by up to 0.68 mean IoU, so automatic detection is crucial.

Signal detection for C4

In C4, we ask models to verify if two images show the exact same individual animal. Because VLMs output a simple Yes or No instead of a similarity score, raw accuracy hides a model’s bias. For example, a model that says Yes to everything gets 50% accuracy without looking at the images. We resolve this using signal detection theory. We track the hit rate HH (saying yes when they are the same) and the false alarm rate FF (saying yes when they are different):

d′=z(H)−z(F),c=−12[z(H)+z(F)](26)d' = z(H) - z(F), \qquad c = -\frac{1}{2}\left[z(H) + z(F)\right] \tag*{(26)}

where zz is the inverse normal cumulative distribution function. We use d′d' to measure how well the model separates the two classes, completely ignoring its answering bias. We use criterion cc to measure that bias. A negative cc means a model prefers saying yes.

d′d'balanced acc.reading
0.050%no discrimination
0.560%weak
1.069%moderate
2.084%strong
3.093%near ceiling

Table 12.

If a model always gives the same answer, the math breaks because zz diverges. We apply a standard log-linear correction, (x+0.5)/(n+1)(x + 0.5)/(n + 1), to the rates unconditionally to prevent this. To get an overall score across species, we pool all the raw counts first

Figure 5. How d′d' and criterion separate discrimination from bias. A negative d′d' indicates the adversarial condition (lower panel), where the decision variable separates the two classes but points the wrong way.

and compute one global d′d'. Sometimes we see a negative d′d'. This means the model says Yes to different individuals more often than to identical ones. We explain this further in Sec. H.

C1 Full Results

We evaluate 20 model configurations on an identical set of 890 000 questions. We test 18 open-weight generalist VLMs and two configurations of the BioCLIP-2 specialist. The tables in this section combine all models unless stated otherwise.

The pooled leaderboard. Table 18 ranks the field by EMsoft\mathrm{EM}_{\mathrm{soft}}.

#ModelImgQuesEMEMsoft_{\text{soft}}wuplin
—BioCLIP-2 kingdom100K890K.743.743.854.836
—BioCLIP-2 tol100K890K.737.737.836.824
1Qwen3-VL-30B-A3B100K890K.503.518.721.702
2Qwen3-VL-32B100K890K.488.505.702.682
3Qwen3-VL-8B100K890K.483.497.707.685
4InternVL3-38B100K890K.455.470.680.661
5Qwen2.5-VL-7B100K890K.445.459.658.638
6Gemma-3-12B100K890K.401.429.646.618
7InternVL3.5-38B100K890K.390.416.635.606
8Gemma-4-12B100K890K.384.394.609.580
9InternVL3-8B100K890K.355.373.575.551
10Qwen2.5-VL-3B100K890K.350.367.594.573
11Llama-3.2-Vision100K890K.338.353.556.529
12InternVL3.5-8B100K890K.325.349.584.552
13Molmo2-8B100K890K.324.349.576.540
14Pixtral-12B100K890K.329.346.566.533
15InternVL3.5-30B-A3B100K890K.317.336.571.544
16Phi-3.5-Vision100K890K.235.245.451.422
17LLaVA-NeXT-Mistral100K890K.203.214.419.395
18LLaVA-NeXT-Vicuna100K890K.202.214.393.368

Table 18. C1 pooled leaderboard, sorted by EMsoft_{\text{soft}}. Every configuration answers the same 890 000 questions over the same 100 000 images, so the rows are directly comparable. The strongest generalist recovers roughly 68% of the specialist pooled accuracy.

Granularity collapse

Table 19 shows the mean exact match score at every taxonomic rank. Model accuracy collapses as we move down the taxonomic tree. When we pool all 20 configurations, exact match falls by a factor of 8.7 from phylum to species epithet. When we look at just the 18 VLMs, it falls by a factor of 20.7. Every single model shows this exact pattern. Two things cause this. First, the number of possible answers grows rapidly from dozens of phyla to thousands of species. The choice simply becomes much harder. Second, specific species appear much less often in training data than entire phyla do. Models have less evidence to learn from. A species-level question demands the sharpest discrimination from the sparsest data, and these two penalties multiply together. This highlights why we include Lin similarity. Agreeing with the truth at a rare, deep clade contributes more information than agreeing on a broad phylum. Yet, models fail exactly on these high-information, fine-grained tasks.

Rank20 models EM20 models step18 VLMs EM18 VLMs stepgap
phylum.748—.725—.023
class.578↓.170.545↓.180.033
order.568↓.010.541↓.004.027
family.421↓.147.382↓.159.039
genus.292↓.129.247↓.135.045
scientific name.165↓.127.123↓.124.042
species epithet.086↓.079.035↓.088.051
common name MCQ.677—.645—.032
common name.085↓.592.039↓.606.046

Table 19. Exact match falls at every step down the hierarchy. The step column gives the change from the row above, in red as the scores fall. The left block pools all 20 configurations, and the right block pools the 18 VLMs. The gap column is the difference between the two, and it widens from .023 at phylum to .051 at species, so the specialist contributes most exactly where the generalists collapse.

Coarse accuracy is inflated

Chordates make up half of the images in C1. Because of this, a model that never looks at the image and just guesses Chordata every time gets a .500 score at the phylum rank. Three models actually score below this blind guess: InternVL3.5-30B-A3B (.486), Qwen2.5-VL-3B (.416), and LLaVA-NeXT-Mistral (.364). A blind rule beats them. When we subtract this blind baseline floor, models recover .450 of the possible headroom at the phylum level, and .494 at the class level. This means models actually do better at class than at phylum in headroom recover. The raw phylum accuracy is inflated partly because the answer pool is so narrow. In contrast, the most common family only covers 6% of the images. The most common species covers just 0.5%. At these specific ranks, a blind model scores near zero. Therefore, scoring high at these ranks requires finer granularity.

Figure 6. Every model’s accuracy falls at every step down the tree. The dashed floor line shows that apparent competence at the phylum level largely comes from guessing common categories, not from actually reading the image.

Recognition exceeds recall

Tab. 20 compares how well VLMs recognize names versus recall them from scratch.

This is a significant effect we see in the benchmark. Past biological VQA works rely heavily on multiple-choice questions. If a benchmark only asks multiple-choice questions, it reports highly inflated scores. Models have the knowledge, but they cannot retrieve it without a hint. These high MCQ scores will immediately crash when users test the models in open-ended real-world use.

Scale is not the bottleneck

Models fail at finer ranks, and scaling it up does not solve this problem. Within the Qwen3-VL family, EM stays practically flat. It moves from .483 at 8B parameters to just .488 at 32B. We see this further when we break scores down by rank. When InternVL3 scales from 8B to 38B, it gains 18.7 points at the class level. But it only gains 6.5 points at the scientific name level. Extra parameters just help the model better map broad categories it already knows. They do not help it distinguish fine-grained species. The real bottleneck is the training data. Rare species just do not appear often enough on the web to train effectively, regardless of how large we make the model.

Figure 7. Every model recognizes the right answer from a list vastly better than it produces the name unprompted.

Difficulty stratification

We group questions into Easy, Medium, and Hard tiers based purely on how often the animal or plant appears in the source catalog. We never define an item as hard just because models fail it. All twenty configurations follow the exact same pattern:

EMEasy(0.497)>EMMedium(0.342)>EMHard(0.190).\mathrm{EM}_{\mathrm{Easy}}(0.497) > \mathrm{EM}_{\mathrm{Medium}}(0.342) > \mathrm{EM}_{\mathrm{Hard}}(0.190).

Every single model follows this order. This proves that a species’ prevalence in the training data genuinely controls how hard it is to classify, completely independent of the model architecture. Sec. E details this setup.

Severity dissociates from accuracy

Severity measures how far a wrong answer lands from the truth, in ranks. We measure that distance through a node, so the model’s answer has to name one. A reply of a small brown bird names no node. We call that answer unresolved and we leave it out of the severity average. Exact match still scores it zero, because a string always compares against a string. Writing TT for the set of nodes in the tree,

sev=E ⁣[d(y)−d(ℓ)  |  y^≠y,  y^∈T](27)\mathrm{sev} = \mathbb{E}\!\left[ d(y) - d(\ell) \;\middle|\; \hat{y} \ne y,\; \hat{y} \in\mathcal{T} \right] \tag*{(27)}

Table 21 gives it for every model at every rank, and Table 22 gives it for every organismal group. The unres column in Table 21 shows the percentage of answers we leave out per model. It runs from .025 to .233 and correlates with exact match at ρ=−0.854\rho= -0.854, so the models that fail to resolve are the models that were going to be wrong anyway. We print it beside severity because a model with a high share is averaged over a smaller sample of its own errors.

Modelphyclsordfamgenspcscicommcqpooled
BioCLIP-2 kingdom1.00†1.00^{\dagger}1.111.832.402.782.882.892.292.722.523
BioCLIP-2 tol1.00†1.00^{\dagger}1.612.222.652.942.962.992.372.722.638
Qwen3-VL-30B-A3B0.530.721.601.842.063.472.482.713.042.472
Qwen3-VL-32B0.380.771.411.782.093.522.482.893.022.459
Qwen3-VL-8B0.610.691.421.802.043.652.462.853.042.480
InternVL3-38B0.650.621.491.902.293.712.823.223.102.614
Qwen2.5-VL-7B0.580.921.371.762.153.772.583.223.092.444
Gemma-3-12B0.190.831.571.972.383.722.933.063.172.574
InternVL3.5-38B0.210.751.601.742.353.782.953.593.122.606
Gemma-4-12B0.991.001.842.042.483.903.073.183.202.763
InternVL3-8B0.460.931.621.912.493.903.053.503.172.692
Qwen2.5-VL-3B0.290.701.051.732.203.862.612.933.142.221
Llama-3.2-Vision0.160.701.431.772.324.052.753.263.182.493
InternVL3.5-8B0.280.911.492.012.604.093.213.623.222.740
Molmo2-8B0.190.971.822.002.534.003.153.533.202.746
Pixtral-12B0.431.001.712.102.623.963.233.693.232.775
InternVL3.5-30B-A3B0.420.721.691.892.503.813.123.633.172.667
Phi-3.5-Vision0.831.311.652.303.124.023.914.013.342.992
LLaVA-NeXT-Mistral0.640.761.862.083.044.193.284.093.322.739
LLaVA-NeXT-Vicuna0.931.072.022.243.664.354.024.183.403.135

Table 21. Mistake severity by rank, with the unresolved share beside it (lower is better). Colors mark the best, second best, second worst and worst over the 18 generalists only. Severity is conditional on being wrong and on resolving, so it is not comparable across models with very different accuracy. †BioCLIP-2 carries the worst phylum severity in the table because it is wrong at phylum only 1.5% of the time, and that residual is the hard part.

GroupEMseverityunresolved
Bryophytes.2463.396.154
Fish.2742.966.109
Gymnosperms.3212.424.176
Reptiles.3232.463.167
Angiosperms.4012.423.101
Amphibians.4042.493.122
Arachnids.4522.479.108
Mammals.4962.483.123
Insects.5282.573.095
Birds.5492.531.097

Table 22. Exact match, severity and the unresolved share by organismal group, pooled over all 20 configurations and sorted by exact match. Unresolved is the percentage of answers that name no node in the tree, so they carry no severity. The three columns do not track each other. Gymnosperms carry the lowest severity and the second highest unresolved share, so their few measurable misses are close …

Severity rises with depth for every model. A wrong phylum answer lands within one rank of the truth for all 18 generalists, from 0.16 to 0.99. A wrong species answer lands three to four ranks up, from 3.47 to 4.35. So a model that misses the species does not miss it by a sibling. It misses it by a family or an order. The species column is also the narrowest, with a spread of 0.88 across the field against 2.05 at genus.

Severity is conditional on being wrong. BioCLIP-2 has the worst phylum severity in the table at 1.00, worse than every generalist except Gemma-4-12B. That is not a defect. BioCLIP-2/kingdom answers phylum correctly 98.5% of the time, so its severity is computed on a residual 1.5% of items, and those are the genuinely hard ones. Llama-3.2-Vision is wrong at phylum 45% of the time and averages 0.16, because most of its errors are near misses inside a large easy set. We therefore read severity within a model across ranks, or between models of similar accuracy, and not as a standalone ranking. Table 21 prints the pooled column beside the per-rank columns for that reason.

Accuracy and severity rank models differently. The rank correlation between EM and severity is ρ(EM, sev)=−0.662\rho(\mathrm{EM},\ \mathrm{sev}) = -0.662, which means they order the models’ performance differently. Qwen2.5-VL-3B is the clearest case of this. It has the best pooled severity in the field at 2.221 and the best severity at order and at family, while ranking 12th of 18 on EM. It is bad at being right and good at being wrong in the right neighborhood, and it moves 11 places between the two orderings. The opposite case is Phi-3.5-Vision, which is second worst pooled at 2.992 and worst at class, family and scientific name. The same signature appears in Sec. F, where Phi is the only model whose C2 severity exceeds 1. It is a model that is wrong confidently in both components. Wu–Palmer and Lin correlate with EM at ρ=0.959\rho= 0.959 and ρ=0.965\rho= 0.965 respectively, so they reorder nothing from the EM rank.

Figure 8. Accuracy and severity measure different things. A model can be wrong often and still miss by a close relative.

Taxonomic revision lag

Biologists often rewrite the taxonomy tree and rename groups. When they do, older textbooks and training data become outdated. We use the gap between soft exact match (EMsoft\mathrm{EM}_{\mathrm{soft}}) and strict exact match (EM) to measure this. Table 23 gives the gap by rank on the left and by organismal group on the right. Our observations are:

RankEMEMsoft_{\mathrm{soft}}gapGroupEMEMsoft_{\mathrm{soft}}gap
class.578.658↑.080Reptiles.323.422↑.100
phylum.748.789↑.041Angiosperms.401.432↑.031
scientific name.165.172↑.007Gymnosperms.321.339↑.018
family.421.425↑.004Bryophytes.246.250↑.004
genus.292.295↑.003Fish.274.277↑.003
order.568.570↑.002Amphibians.404.407↑.002
species epithet.086.086—Arachnids.452.453↑.001
common name.085.085—Birds.549.550↑.001
common name MCQ.677.677—Insects.528.528↑.001
all ranks.400.416↑.015Mammals.496.498↑.001

Table 23. Taxonomic revision lag, pooled over all 20 models. The gap is EMsoft_{\mathrm{soft}} − EM, so it is always a gain and we mark it with a teal arrow. Both panels are sorted by gap. The two panels line up. Reptiles carry the largest group gap and Reptilia dissolved at class rank, which is where the largest rank gap sits. Angiosperms and Gymnosperms carry the next two and their revision moved the phylum from Magnoliophyta to Tracheophyta, which is the second largest rank gap. A dash means the gap rounds to zero.

  • Class carries the largest gap at .080. Reptilia is a class-rank name and the backbone dissolved it into four modern classes. Reptiles carry the largest group gap at .100, three times the next group. A model answering Reptilia for a snake is repeating textbook Linnaean taxonomy, and our strict backbone marks it wrong.

  • Phylum carries the second largest at .041. The plant revision moved the phylum from Magnoliophyta to Tracheophyta. Angiosperms and Gymnosperms carry the second and third largest group gaps at .031 and .018.

  • Everything below order is flat. From family down the gap never exceeds .004, and at the species epithet, the common name and the MCQ it rounds to zero. Nothing was renamed there, so there is nothing to credit.

The gap is not a general leniency in the metric. It appears only where the taxonomy moved.

We verify every soft credit against GBIF with a separate audit script. All of roughly 130 000 soft credits resolve to a valid old-to-new substitution and none is flagged for review. So this EM-soft gap measures how old a model’s training data is, not an actual failure to identify the animal.

Cross-group generality

We check if models that classify birds well also classify fish well. They do. We rank the models for each of the 10 organism groups. We then compare these rankings using Spearman rank correlation. We summarize the resulting 10×1010 \times10 matrix by its mean off-diagonal value. Table 24 gives the full matrix.

AmphAngioArachBirdsBryoFishGymnoInsectMammReptmean
Amphibians1.00.94.82.90.92.88.39.89.95.98.852
Angiosperms.941.00.87.88.96.89.49.91.92.94.867
Arachnids.82.871.00.90.87.81.41.94.83.83.809
Birds.90.88.901.00.91.87.42.97.91.89.850
Bryophytes.92.96.87.911.00.94.46.95.89.91.868
Fish.88.89.81.87.941.00.37.91.83.88.820
Gymnosperms.39.49.41.42.46.371.00.45.46.45.433
Insects.89.91.94.97.95.91.451.00.88.90.867
Mammals.95.92.83.91.89.83.46.881.00.95.847
Reptiles.98.94.83.89.91.88.45.90.951.00.859

Table 24. Spearman rank correlation between organismal groups. The mean column averages the nine off-diagonal cells in each row. Red marks the Gymnosperms row and column, which hold the lowest value in every row of the table without exception, and Arachnids carries the next lowest mean, but almost 2× Gymnosperms. The whole matrix averages 0.807. If we drop Gymnosperms, it jumps to 0.901, so one group costs the benchmark almost a tenth of its cross-group agreement.

Our observations are:

  • Taxonomic ability is mostly one-dimensional. If we ignore Gymnosperms, every pairwise correlation sits between 0.81 and 0.98. Models that rank highly on amphibians also rank highly on mammals, fish, and birds. A single underlying taxonomic ability drives the leaderboard. The specific clade barely matters. We use this strong correlation as our baseline when we analyze cross-task abilities in Sec. J.

  • Gymnosperms break the pattern. Every correlation involving Gymnosperms collapses to roughly 0.4. This is not noise. The Gymnosperm label space mostly consists of very similar-looking conifer trees. The specialist model handles this highly repetitive visual space very differently than generalists do. This difference scrambles the leaderboard for this group alone. We report it instead of smoothing it away.

Figure 9. Model rankings remain nearly identical across nine of the ten organism groups. Gymnosperms are the single exception.

Difficulty tiers

We rank question difficulty by how common the target taxon is. We count how many times it appears in the TreeOfLife-200M database. Let n(t)n(t) be its image count, and Tr\mathcal{T}_r be all taxa at rank rr. We calculate its prevalence percentile as:

P(t)=∣{u∈Tr:n(u)≤n(t)}∣∣Tr∣.(28)P(t) = \frac{\left|\left\{u \in\mathcal{T}_r : n(u) \le n(t)\right\}\right|}{\left|\mathcal{T}_r\right|}. \tag*{(28)}

If a taxon sits in the top third (P(t)>0.67P(t) > 0.67), we label it Easy. If it sits in the bottom third (P(t)≤0.33P(t) \le0.33), we label it Hard. We label the remainder Medium.

Non-English common names.

The GBIF Backbone vernacularName field is multilingual by specification. TreeOfLife-200M inherits the field unchanged, and the download surfaces no per-record language indicator. So a non-trivial share of our common-name questions arrive with a Swedish, Danish or Norwegian ground truth. It breaks the questions two ways. A model that recognizes the organism and answers Moor frog is marked wrong against a ground truth of åkergroda, so the item stops measuring recognition. And a foreign string sitting in an otherwise English option set is an odd one out, so a model can pick it without looking at the image. The second failure is the worse one, because it turns a visual item into a textual one.

Scientific nameAs shippedLang.Resolved
Juniperus communisensv/daCommon juniper
Picea abiesgransvNorway spruce
Pinus sylvestristallsvScots pine
Rana arvalisåkergrodasvMoor frog
Rana arvalisSpidssnudet frødaMoor frog
Bufotes viridisgrönfläckig paddasvEuropean green toad
Anguis fragiliskopparödlasvSlow worm

Table 25. Contaminated ground truth labels and their resolutions.

We resolve each affected taxon through three stages and each taxon leaves the pipeline as soon as one stage settles it. First we triage offline. A GBIF language tag of en accepts a name, a name carrying non-ASCII letters is rejected before any other test, and otherwise we score the fraction of tokens that appear in an English wordlist. We reject before we accept here on purpose. A foreign name wrongly accepted poisons the answer key, while a false rejection only costs one lookup.

Second, we query authorities in order, taking the first clean English candidate. iNaturalist gives a preferred common name, GBIF gives English vernaculars after following synonyms to the accepted name, and Wikipedia’s lead sentence is the last resort. Every candidate passes a gate that rejects rank words, taxonomic qualifiers and echoes of the scientific name, so a doubtful candidate is deferred rather than accepted.

Third, a curator resolves the residual by hand with a cited source. These are obscure invertebrates and newly described species with no English name in any authority. Because the stages are sequential and disjoint, no taxon appears in two tiers, and the merge produces zero conflicts.

Difficulty Stratification

Every C1 question carries a difficulty tier derived strictly from the taxonomy itself, rather than from empirical model performance. We compute this difficulty once over the union of all answer keys and propagate it identically to every model’s prediction file, ensuring all models are evaluated against an identical stratification.

Design principles

Difficulty is an intrinsic property. We compute tiers solely from the taxonomy and the answer key. No prediction file influences any tier, which prevents the circularity of defining an item as hard simply because models struggle with it.

Unified computation. We assemble a corpus of unique group, image, and question triples across all matched files prior to scoring. We then compute one tier per item and write it to all files containing that item. A model with missing predictions cannot inadvertently shift a boundary and receive a different stratification from its peers.

Taxon-deterministic tiers. Two items sharing a target taxon always receive the same tier. Because a taxon never straddles a tier boundary, per-tier accuracy reliably reflects performance over a set of taxa rather than an arbitrary partition of individual items.

Exhaustive, audited resolution. Every item resolves to a specific node in the taxonomy. We record the resolution mechanism per item, allowing the proportion of non-direct resolutions to be explicitly reported.

Two axes for two formats

Because the two question formats measure distinct competencies, we evaluate them along two different axes. Table 26 gives the axis used for each format.

FormatAxisQuestion it answers
Open recallPrevalenceHow much of the visual world does this taxon cover?
4-way MCQConfusabilityHow close is the nearest wrong option in the tree?

Table 26. The difficulty axis used for each question format. Prevalence is unsuitable for a choice item: recognizing a common organism among three close relatives is harder than recognizing a rare organism among three distant relatives.

The prevalence tier

For a given target taxon tt, let c(t)c(t) denote the number of images whose lineage passes through taxon tt. Exposure is the number of catalog images beneath it:

c(t)=∣{i:t∈lineage⁡(i)}∣,x(t)=log⁡10(c(t)+1).(29)c(t) = \left|\{i : t \in\operatorname{lineage}(i)\}\right|,\qquad x(t) = \log_{10}(c(t) + 1). \tag*{(29)}

We report exposure on a logarithmic axis because catalog abundance is heavily right-tailed. The most-photographed species appear in 10510^{5} to 10610^{6} images, whereas the rarest appear in only a handful—a span of five to six orders of magnitude. Difficulty does not scale linearly with this count: the jump from 10 to 100 images represents the same multiplicative change in rarity as the jump from 10 000 to 100 000, and this multiplicative scale better reflects the volume of visual evidence a model has plausibly encountered. The base-10 logarithm maps each order of magnitude to one unit on the axis, while the unit pseudocount ensures the transformation remains defined for taxa with empty subtrees. The tertile cut below is rank-based and therefore invariant to this monotone transform; x(t)x(t) is the quantity we report and use wherever a continuous prevalence covariate is required, and on a linear axis that display would collapse almost all taxa into an indistinguishable low-count mass.

We then divide taxa into tertiles within a given reference cell. Distinct taxa are sorted by exposure (with taxon names breaking ties), and each taxon is assigned an item-weighted mid-rank percentile:

p(t)=cum⁡(t)+12n(t)Ncell(30)p(t) = \frac{\operatorname{cum}(t) + \frac{1}{2}n(t)}{N_{\mathrm{cell}}} \tag*{(30)}

where cum⁡(t)\operatorname{cum}(t) is the sum of item counts for rarer taxa, n(t)n(t) is the number of items targeting tt, and NcellN_{\mathrm{cell}} is the total item count in the cell. A taxon with p(t)<1/3p(t) < 1/3 is Hard, one with 1/3≤p(t)<2/31/3 \le p(t) < 2/3 is Medium, and the remainder are Easy; every item inherits the tier of its target taxon.

Two structural requirements dictate this formulation. First, a tier must represent a set of taxa rather than a mere partition of items; thus, each taxon must receive exactly one percentile. Second, the three strata must contain comparable item mass to preserve statistical power for per-tier accuracy estimates. Assigning

CellDistinct taxaOutcome
(Birds, phylum)1, Chordataescalate
(phylum, all groups)a handfuldegenerate
(Birds, class)1, Avesescalate
(class, all groups)manytertiles, global

caption: The escalation protocol for resolving thin cells prevents the fabrication of non-existent contrasts. Because every bird belongs to the phylum Chordata, and the entire benchmark spans only a few phyla, there is genuinely no rare-versus-common phylum contrast to measure within birds.

:::

each taxon the midpoint of its item-mass block satisfies both criteria. Using the midpoint rather than the leading edge prevents the boundary-splitting failures typical of naive cuts on sorted lists, and the item-count weighting guarantees balanced strata.

Reference cells and thin cells

We define reference cells by organismal group and rank class. The rank class consolidates three species-level wordings (epithet, scientific name, and common name) to ensure difficulty reflects the organism itself, not the phrasing of the question.

Stratifying within a rank prevents the confounding effect where higher ranks inherently possess larger subtree counts (e.g., a class always contains more images than any of its constituent families). Stratifying within a group ensures that rare is defined relative to the specific peers a model is tasked with distinguishing.

A tertile split is only statistically meaningful when a cell contains enough distinct taxa. However, some cells are structurally thin. For instance, within a single group, every phylum answer might be identical, and class answers might span only two or three values.

If a cell contains fewer than six distinct taxa, we escalate the stratification, re-cutting against that rank class pooled across all groups. Because exposure is already a global metric, this pooled cut provides a coherent reference population. A cell is only labeled as degenerate if the pooled cell also proves too thin. Table 27 details the escalation protocol for a thin cell.

The degenerate label marks a structural reality, not a pipeline failure. It indicates items for which no meaningful prevalence contrast exists. These items still possess resolved targets and exposure counts, but are excluded from prevalence-stratified analyses because sorting two or three taxa into three tiers yields no actionable insight. All escalations and degenerate cell designations are recorded in the run log.

The confusability tier

Let d(⋅)d(\cdot) denote node depth, with the root at depth 0, and let DD be the set of distractors for an item (in ∣D∣=3|D| = 3). For a ground-truth node gg and each distractor j∈Dj \in D, the nearest-distractor distance (NDD) is

dj=d(g)−d(LCA⁡(g,j)),NDD⁡=min⁡j∈Ddj(31)d_j = d(g) - d(\operatorname{LCA}(g,j)), \qquad\operatorname{NDD} = \min_{j \in D} d_j \tag*{(31)}

NDD is measured in edges rather than named ranks. An NDD of at most 1, in practice exactly 1, since two distinct nodes differ by at least one edge is Hard, an NDD of 2 is Medium, and an NDD of at least 3 is Easy. For the species-level targets that dominate the benchmark these correspond to a congeneric, a con-familial, and a more distant nearest distractor respectively; because vernacular targets may sit at coarser ranks (Sec. E.7), we additionally emit the rank at which the nearest distractor meets the answer, so that an edge distance can always be read back as a rank.

The depth of the lowest common ancestor (LCA) serves as the standard metric for confusion severity. Two taxa meeting only at the root are maximally dissimilar, whereas two meeting at the genus are closely related. Taking the minimum distance across all three distractors enforces a weakest-link model for multiple-choice items: a single genus-level distractor renders the entire question Hard, even if the other two distractors belong to a different phylum, because the model must still execute fine-grained discrimination to exclude that near option. Using the maximum or mean distance would allow easy distractors to artificially mask a hard one.

To complement this tier, we introduce a continuous confusability score where each additional rank of separation halves the credit:

confusability⁡=1∣D∣∑j∈Dexp⁡(−ln⁡2⋅dj)=1∣D∣∑j∈D2−dj.(32)\operatorname{confusability} = \frac{1}{|D|}\sum_{j \in D}\exp(-\ln2 \cdot d_j) = \frac{1}{|D|}\sum_{j \in D}2^{-d_j}. \tag*{(32)}

This geometric decay matches the intuition that taxonomic proximity scales multiplicatively. Averaging the contributions provides a comprehensive profile summary that complements the NDD floor summary. Analyses requiring a continuous covariate should use this confusability score, while categorical analyses use the three-level NDD tier.

Worked example. Consider an item whose answer is Zootoca vivipara at depth 6, with three distractors: one that shares its family, one that shares only its class, and one from a different phylum (Table 28). The NDD is 2 (Medium), yielding a confusability score of 0.110. If the first distractor is replaced by a congener, the LCA moves to the genus level (depth 5), so the distance djd_j drops to 1, and its contribution rises to 0.500. The NDD drops to 1, pushing the item into the Hard tier, and confusability increases to 0.193. This demonstrates how a single close negative appropriately flips the tier under the weakest-link minimum constraint.

Target resolution

Every item resolves to exactly one node. Because each image’s answer key provides its complete lineage, resolution does not rely on a single answer string being independently discoverable. We apply a four-step protocol and record the step that successfully resolves the target. Table 29 outlines the four steps alongside examples.

StepRuleExample
exactthe answer names a node at the asked rankLacertidae as family
vernaculara common name through the vernacular indexOrb weavers to Araneidae
lineagethe asked rank read off this image’s lineagevivipara through the image
anchorlowest resolvable node in the lineagean order question with a null order

Table 29. The four-step resolution protocol. The lineage step provides essential robustness; for example, a question asking for an organism’s class resolves through the organism’s full metadata rather than just the isolated string.

DistractorLCAdepthdjd_j2−dj2^{-d_j}
same familyfamily420.250
same classclass240.063
other phylumroot060.016

caption: Worked example for a ground truth node Zootoca vivipara at depth 6. The nearest distractor sits two edges away, so the item is Medium under the weakest-link NDD rule and scores 0.110 on the continuous axis.

:::

StepRuleExample
exactthe answer names a node at the asked rankLacertidae as family
vernaculara common name through the vernacular indexOrb weavers to Araneidae
lineagethe asked rank read off this image’s lineagevivipara through the image
anchorlowest resolvable node in the lineagean order question with a null order

Vernacular names and rank heterogeneity

Common names do not function as a strict parallel nomenclature for species. In biodiversity corpora, they denote taxa at whatever rank a naturalist finds useful, and this rank varies heavily by clade. While birds and mammals almost universally possess species-level vernaculars, arachnids and bryophytes frequently do not; their familiar names often refer to families or orders (e.g., Orb weavers maps to Araneidae, Long-jawed orb weavers to Tetragnathidae, and Daddy longlegs to an entire order). A resolution framework that naively maps one common name to one species would systematically misclassify items across entire phyla.

To address this, we derive the vernacular index dynamically from the corpus. Each image contributes a pair comprising its common-name answer and its resolved species. A name observed alongside a single species resolves directly to that species. A name observed across multiple species resolves to their lowest common ancestor, accurately reflecting the broader taxon the name denotes in practice.

This data-driven approach yields three key properties. First, rank is inferred rather than strictly asserted: the index discovers that a name is family-level because the corpus uses it as such. Second, exposure accurately matches the question; a family-level name resolves to a family node and inherits the family’s broader prevalence, correctly reflecting that identifying a spider family is easier than identifying a specific species. Third, distractors resolve successfully, as MCQ distractors are drawn from the same pool of common names as the correct answers.

Consequently, the species-level reference cell inherently includes a small number of coarser-rank targets that carry larger exposures. This heterogeneity is deliberate, capturing a genuine difference in task difficulty while remaining strictly within the phrasing constraints of the original question.

Output columns. We append 13 metadata columns to each processed file: the resolved target taxon, its rank, the reference cell, the resolution mechanism, the exposure count, the log⁡10\log_{10} exposure, the prevalence percentile, the prevalence tier, the nearest-distractor distance (NDD), the taxonomic rank at which the NDD meets the answer, the continuous conspecificity score, the MCQ tier, and a unified difficulty column (which applies the MCQ tier for choice items and the prevalence tier for open-recall items).

C2 Full Results

We report CAGS and EM metrics alongside one another throughout this section. EM serves as a conservative anchor, while CAGS acts as a finely-grained diagnostic instrument. Their agreement validates the overall model ranking, and their divergences help localize specific model behaviors.

cags leaderboard

Table 30 ranks the field by CAGS. Performance across the evaluated models ranges from .378 to .593, demonstrating an absence of ceiling saturation, an essential characteristic of a robust benchmark. The graded metric immediately surfaces three behavioral anomalies, which we expand upon below.

#ModelEMGSrawGS_{\mathrm{raw}}CAGS95% CIsev
1Qwen3-VL-32B.570.687.593.592–.5940.946
2Qwen3-VL-30B-A3B.538.665.569.568–.5700.933
3InternVL3-38B.535.665.568.567–.5690.929
4Gemma-4-12B.526.651.554.553–.5550.941
5Qwen3-VL-8B.528.654.553.551–.5550.947
6InternVL3.5-38B.528.648.548.546–.5490.959
7Molmo2-8B.520.634.533.532–.5340.974
8Qwen2.5-VL-7B.497.635.529.528–.5300.936
9InternVL3.5-30B.501.631.524.523–.5250.954
10InternVL3-8B.486.619.507.506–.5080.959
11InternVL3.5-8B.461.612.505.504–.5060.919
12Gemma-3-12B.468.605.487.486–.4880.965
13Qwen2.5-VL-3B.441.586.461.460–.4620.964
14Pixtral-12B.414.565.439.438–.4400.956
15LLaVA-NeXT-Vicuna.367.541.419.418–.4200.918
16Phi-3.5-Vision.415.535.405.403–.4061.018
17Llama-3.2-Vision.376.527.382.380–.3830.990
18LLaVA-NeXT-Mistral.376.517.378.377–.3790.997

Table 30. C2 leaderboard over 824 104 questions per model. Intervals are 95% bootstrap over B=1000B = 1000 resamples of the per-item CAGS. Adjacent ranks separated by less than 0.001 are statistical ties. Phi-3.5-Vision is the only model with severity above 1 and the only one whose CAGS falls below its EM.

EM leaderboard and format diagnostics

Tab. 31 provides the EM scores alongside three format-specific diagnostics.

Parse rates remain uniformly at or above 98.9%, ensuring no model is unfairly penalized for malformed formatting. However, two choose A values stand out as strong indicators of position bias. Llama-3.2-Vision selects option A nearly two-thirds of the time, revealing a prior over answer position rather than content. LLaVA-NeXT-Mistral selects A approximately one-third of the time. Qwen2.5-VL-7B shows the highest letter–text mismatch rate in the field at 1.17%; while small in absolute terms, it is the only model where the benchmark’s letter-against-text arbitration policy produces a measurable effect.

Per-category cags

Tab. 32 breaks down CAGS by question category. The naming category shows the widest performance spread (ranging from .198

ModelAccMCQOpenparsemism.choose A
Qwen3-VL-32B57.3157.6351.96100.00.0019.15
Qwen3-VL-30B-A3B54.0854.8241.53100.00.0324.30
InternVL3-38B53.7054.4241.72100.00.0217.08
Qwen3-VL-8B52.9853.9237.11100.00.0414.56
InternVL3.5-38B52.8854.3927.57100.00.0222.85
Gemma-4-12B52.7155.0613.3398.900.0117.00
Molmo2-8B52.0954.5410.94100.00.0221.41
InternVL3.5-30B50.2251.4629.49100.00.0227.41
Qwen2.5-VL-7B49.9950.0449.0599.931.1713.98
InternVL3-8B48.6949.6832.1499.850.1617.87
Gemma-3-12B46.9648.4621.79100.00.0214.63
InternVL3.5-8B46.1547.6720.68100.00.0115.35
Qwen2.5-VL-3B44.4444.4344.48100.00.0510.77
Pixtral-12B41.5142.7221.21100.00.1020.83
Phi-3.5-Vision41.4843.733.81100.00.0617.50
LLaVA-NeXT-Mistral37.6539.693.36100.00.2133.96
Llama-3.2-Vision37.5538.0429.3399.960.0263.26
LLaVA-NeXT-Vicuna36.7138.732.83100.00.1917.98

caption: C2 EM in percent, alongside three format diagnostics. The choose A metric tracks the frequency of multiple-choice answers equal to option A (against a uniform expectation near 20%). Every model’s MCQ score exceeds its open-recall score, often by tens of percentage points.

:::

to .731), indicating that taxonomic nomenclature remains a primary differentiator among models. Conversely, the morphological attributes category is the most compressed (.454 to .645), suggesting that basic attribute extraction is a broadly shared competence. Phi-3.5-Vision’s score of .154 on anatomical localization represents the lowest single category score in the evaluation.

Per-family cags

Tab. 33 details cags across all models and families, while Tab. 34 provides the EM scores on the identical item partition.

Family difficulty

Tab. 35 orders 10 question families by mean cags across the field.

The recognition–recall gap

Tab. 36 contrasts the two naming formats for the models exhibiting the widest and narrowest performance gaps. InternVL3.5 models show the largest gaps (from .51 to .57). They recognize items well, but they struggle to generate names. On the other hand, the Qwen models show the smallest gaps. They generate names as well as they recognize them. Llama-3.2-Vision scores .235. It ranks 17th overall. However, its open common-name cags of .624 is the second highest. The model names birds very well. Its bad multiple-choice score comes from position bias, not poor knowledge. We only see this by looking at cags and format metrics together.

Three anomalies the graded metric surfaces

Position bias. Llama-3.2-Vision picks option A 63.26% of the time. This is about 3.2× the normal rate. LLaVA-NeXT-Mistral picks option A 33.96% of the time. Llama gets many correct answers just by guessing A. This shows we need a score that fixes position bias. We also must report how often models choose A.

ModelAnat. loc.Morph.SalienceNaming
Qwen3-VL-32B.4754.6441.5839.7310
Qwen3-VL-30B-A3B.4525.6313.5582.6584
InternVL3-38B.4132.6451.5621.6691
Gemma-4-12B.4306.6205.5915.3742
Qwen3-VL-8B.4411.6244.5309.6429
InternVL3.5-38B.3747.6379.5595.5363
Molmo2-8B.4741.6429.5027.3301
Qwen2.5-VL-7B.3556.5800.5409.7127
InternVL3.5-30B.3504.6093.5367.5391
InternVL3-8B.3462.5844.5081.5813
InternVL3.5-8B.3722.6028.4953.4572
Gemma-3-12B.4103.5506.4657.5113
Qwen2.5-VL-3B.3890.5005.4272.6615
Pixtral-12B.3199.4950.4471.4654
LLaVA-NeXT-Vicuna.3640.4535.4571.1977
Phi-3.5-Vision.1541.5335.4540.2175
Llama-3.2-Vision.2862.4831.3168.5203
LLaVA-NeXT-Mistral.2587.4843.3722.2383

caption: cags by category. Colors mark the best, second best, second worst and worst model in each column. The category leaders vary across domains: InternVL3-38B leads morphological attributes, Gemma-4-12B leads relative salience, and Qwen3-VL-32B leads anatomical localization and naming. Molmo2-8B is second on anatomical localization at .4741, which is .0013 behind a model four times its size. A single aggregated scalar masks these localized strengths.

:::

Confidently wrong. Phi-3.5-Vision is the only model with a severity over 1 (at 1.018). It is also the only model where cags (.405) is lower than EM (.415). For all other models, cags is equal or higher. This happens because we give partial credit for near misses. Phi is different. Its wrong answers are worse than random guesses. This is very clear on visibility_parset. Here, its cags drops to .108 and severity rises to 1.088. A severity over 1 shows why we need cags. EM cannot show this problem.

The grounding outlier. Molmo2-8B gets the highest score on visibility_side (.746) and marker_localization (.743). It beats much larger 30B and 38B models here. But, it does poorly on naming (.330). It ranks 7th overall. This low rank hides a major strength. Molmo2-8B is the only model in our test that uses explicit pointing training.

cags re-ranks where it matters

Tab. 37 details every model whose ranking shifts between the two metrics.

Because no swap shifts a model by more than three positions, and the top-tier rankings remain undisturbed, cags refines the exact-match ranking rather than upending it. This property ensures the graded metric is both trustworthy and diagnostically rich, rather than merely divergent.

Laterality

We isolate the items where a model successfully identifies the correct anatomical part. Then we evaluate its ability to determine the correct laterality (left versus right). 17 of the 18 models perform at or below chance, with 9 falling significantly below. Mirror confusion ranges from 21.8% to 54.0% (against a 20% uniform baseline), and this error rate correlates heavily with overall model strength (ρ=.75\rho= .75). This combination of factors rules out two potential explanations. First, the failure is not due to an inability to locate the part, as we condition on successful part identification. Second, the models are not simply guessing, as random guessing would drive the rate toward the uniform baseline, not significantly below

ModelMCQOEgap
InternVL3.5-30B.869.301.568
Phi-3.5-Vision.716.177.539
InternVL3.5-38B.920.389.531
InternVL3-8B.884.374.510
InternVL3.5-8B.783.277.506
Gemma-4-12B.772.285.487
InternVL3-38B.948.495.453
Qwen3-VL-32B.959.619.340
Qwen2.5-VL-7B.962.627.335
Llama-3.2-Vision.859.624.235

Table 36. Recognition–recall gap for taxonomic questions.

ModelCAGS rankEM rankΔ\DeltaWhy
Gemma-4-12B46+2near misses earn
LLaVA-NeXT-Vicuna1518+3misses land close
Qwen2.5-VL-7B89+1
InternVL3.5-8B1112+1
Phi-3.5-Vision1615−1confidently wrong
LLaVA-NeXT-Mistral1816−2misses land far
11 models unchanged—0two metrics concur

Table 37. Instances where CAGS and EM disagree. The signature shift is LLaVA-NeXT-Vicuna surpassing Phi-3.5-Vision. While Vicuna achieves fewer EMs, its misses consistently land closer to the truth; thus, a closeness-aware, chance-corrected metric appropriately ranks it higher.

it. Instead, stronger models systematically isolate the correct part and then actively choose the mirrored side. The notable exception is Molmo2-8B at .689, reflecting its unique training with explicit pointing supervision. Laterality exhibits the lowest loading coefficient on the shared cross-component factor (.154), consistent with the premise that it measures a capability largely disjoint from the rest of the benchmark tasks.

We also acknowledge a corresponding limitation in the metric. The CAGS anatomy kernel assigns a high similarity score (.883) between the left wing and right wing because they are spatially proximal in the structural layout. Consequently, the kernel over-credits the exact laterality confusion detailed in this analysis. For this reason, we compute the laterality figures above using EM rather than CAGS. The human validation study (see § 1) confirms that annotators correctly reject these mirrored items, highlighting the limitation in the kernel’s spatial credit assignment.

Figure 10. Color terms are compared perceptually rather than lexically; thus, predicting purple instead of blue receives significantly more partial credit than predicting yellow instead of blue.

Figure 11. Partial credit elevates every model above the identity line, with one exception. Phi-3.5-Vision falls below the line because its prediction errors land further from the ground truth than a random distractor.

C3 Full Results

Every task carries an image-blind floor, which we report in the same table as the score. Multiple tasks fall below their floor, which no accuracy-only report would reveal.

Per-task summary

Table 38 gives the best model and the floor for each of the eight tasks.

TaskBest modelScoreFloor
CountingInternVL3-38B.799modal count .501
ViewpointQwen3-VL-32B.358majority .520, κ\kappaVAS 0
Absolute distanceQwen3-VL-32B.858per-type constant .410
Relative distanceGemma-4-12B.443chance .250
Relative countingQwen3-VL-32B.403chance .250
LocalizationQwen3-VL-32B.866mean box .415
Relative positionQwen3-VL-32B.550majority .125
OCR (control)Qwen3-VL-30B.987constant string .349

Table 38. Per-task headline. Counting is macro-CAS, viewpoint is κ\kappaVAS, absolute distance is within-plate Spearman ρ\rho, relative distance and relative counting are balanced accuracy, localization is mean IoU, relative position is accuracy, and OCR is suffix character accuracy.

Counting

Counting runs on all ten 500-image subsets. The floor is 0.501 on the graded metric and 0.089 on EM, which makes the entire measurable band on the graded metric 0.30 wide and places one of eighteen models within 0.013 of its own floor. Models average .695 macro-CAS against that floor of .501. An average of .695 means the typical answer is off by about 30% of the true count. InternVL3-38B leads at .799. EM, however, does not exceed .242 for any model. So three quarters of the answers miss by at least one animal. EM falls between .066 and .239 against a constant-predictor floor of .089, so several models fall below the floor on EM while sitting well above it on the graded score. We keep both columns for that reason. No model counts 180 bees exactly, so on dense images, EM collapses toward the modal-answer rate and stops discriminating.

The two-panel penalty. We build counting items on three canvas tiers. The Easy tier shows one image filling the canvas. The Medium tier splits the canvas into two panels drawn from different datasets. The Hard tier splits it into two panels drawn from the same dataset. The encoder budget is fixed across tiers, so a target occupies about half the area in Medium and Hard that it occupies in Easy. We score each model on each tier and take the difference between consecutive tiers. Table 39 reports three numbers per contrast. The drop is the mean of those 18 per-model differences, written so a positive value means the harder tier costs accuracy. σ\sigma is the standard deviation of the same 18 differences, so it says how much the effect varies from model to model. The last column counts the models whose own drop is positive. The first row holds for every model without exception, so the panel split is a real and universal effect. The second row holds for a third of models and its mean is within noise, so dataset homogeneity across panels is not what makes the task hard.

ContrastEffectσ\sigmaHolds for
easy to medium+.064.03718 of 18
medium to hard−.005.0276 of 18

Table 39. Cutting 50% the area a target occupies costs every model accuracy. Drawing both panels from the same corpus makes no further difference.

Rank correlation as a shortcut detector. We report the Spearman correlation between predicted and true counts per corpus. A model that answers a constant scores exactly zero, whatever its CAS. A model whose CAS is high and whose ρ\rho is near zero is not counting. It is answering near the modal count and collecting graded credit for proximity. This column is what separates counting from guessing, and neither CAS nor EM can do it alone.

Viewpoint

Viewpoint has ten orientations and a skewed prior, so the three floors differ by a factor of three. A constant right predictor scores EM = 0.520, VAS = 0.534, macroVAS = 0.170 and κ\kappaVAS = 0.00. We report κ\kappaVAS as the headline because it is the only column a constant answer cannot inflate. 17 of 18 models fall below the majority-class floor on EM. The best κ\kappaVAS in the field is .358, shared by Qwen3-VL-32B and Molmo2-8B. Table 40 gives the six models whose rank moves most between the two columns.

ModelEMrankκ\kappaVASrankshift
Molmo2-8B.22810.3581.5+8.5
Qwen2.5-VL-7B.05015.1827.5+7.5
Qwen3-VL-32B.4177.3581.5+5.5
InternVL3.5-38B.4832.2086−4.0
InternVL3.5-8B.4563.5.11710−6.5
Phi-3.5-Vision.4563.5.08311−7.5

Table 40. The six largest rank shifts between EM and κ\kappaVAS on viewpoint. Tied models share the average of the positions they span. InternVL3.5-8B and Phi-3.5-Vision both score EM = .456 and fill positions 3 and 4, so both read 3.5. Molmo2-8B and Qwen3-VL-32B both score κ\kappaVAS = .358 and fill positions 1 and 2, so both read 1.5. A shift is the difference between two midranks and is fractional for the same reason. Raising models place a large share of answers in the over class, which means they name a compound direction where the annotation records a cardinal one. EM scores every one of those as zero.

Molmo2-8B and Qwen2.5-VL-7B each place 62.9% of answers in the over class. Phi-3.5-Vision places 0.7%. So the reordering is not noise. It is an error-composition difference that EM cannot see.

Validating the rubric. The rubric grades a viewpoint answer by how many 45-degree steps separate it from the ground truth. Full credit at zero steps, partial credit at one, less at two, none at the opposite. We check these step weights two ways. First, we re-score the same predictions with a symmetric angular error on the ring. This construction shares none of the rubric’s design decisions. It maps each viewpoint to a bearing and takes the absolute circular difference, so there is nothing to tune. Tab. 41 works one ground truth through both.

GTAnswerstepsangleEM
leftleft00°1
leftfront-left145°0
leftfront290°0
leftright4180°0

caption: One ground truth (GT) viewpoint under both scorers.

:::

The rubric reads the steps column and the angular metric reads the angle column. EM reads neither and gives the last three rows the same zero. Labels that sit off the ring carry no bearing, so angular check covers the ring classes only, which is why we use it as a check and not as the headline.

The over class is the second row of that table. A model answers front-left where the annotation records left. It has named the correct side and added a component the annotation does not carry. EM scores that zero, the same as right. The rubric and the angular metric both place it one step away. That single row is what separates Molmo2-8B, which lands there on 62.9% of its answers, from Phi-3.5-Vision, which lands there on 0.7%, and it is why the two models move eight and a half ranks apart between the columns of Table 40. The two constructions agree on the model ordering. Second, we sweep the rubric weights and re-rank under each setting. The ordering is nearly invariant to them, so the viewpoint result does not rest on the particular credit we assign to one step.

Absolute distance

This is the sharpest below-floor result in the benchmark. Answering one fixed size per measurement type would score acc@20 = 0.410. 15 of 18 models fall below that. Only Qwen3-VL-32B at .622, Qwen3-VL-30B-A3B at .528 and InternVL3.5-38B at .503 beat it. The predictions are well-formed numbers in the right units with far too little spread across specimens. Mean relative error across the field is 0.374.

The compression index. We define κ\kappa as the ratio of predicted spread to gold spread within a measurement type. A model answering from one remembered size has κ≈1\kappa\approx1. Three models sit below κ=0.20\kappa= 0.20, which are Qwen3-VL-32B at 0.085, Qwen3-VL-30B-A3B at 0.168 and InternVL3.5-38B at 0.189. These are the same three models that beat the floor.

Ranking and calibration dissociate. We compute acc@20 a second time after removing each model’s per-type bias. Tab. 42 separates models that measure from models that calibrate.

Modelρwithin\rho_{\mathrm{within}}acc@20calibdiagnosis
InternVL3.5-8B.703.263.509measures, no calibration
Gemma-4-12B.753.385.512measures, no calibration
Qwen3-VL-8B.682.303.453measures, no calibration
Qwen3-VL-32B.858.622.614measures and calibrates
Qwen2.5-VL-3B.081.194.306neither
LLaVA-NeXT-Vicuna−.023.217.326neither

caption: Measurement and calibration come apart. InternVL3.5-8B orders the specimens well at ρwithin=.703\rho_{\mathrm{within}} = .703 but sits below the floor on precision at .263. Removing its per-type bias lifts it to .509, from below the floor to above it.

:::

This is the reason we report ρwithin\rho_{\mathrm{within}} as the headline rather than acc@20. A model can read the image and still answer on the wrong scale. Accuracy alone calls that failure and ranking alone calls it success.

Localization

Tab. 43 gives the full leaderboard with the mean gold box inserted at its rank. The convention detection is crucial. InternVL3-38B

#ModelIoUall_{\mathrm{all}}a.25a.50a.75parseconvention
1Qwen3-VL-32B.866.995.984.8771.000n1000 xyy
2Qwen3-VL-8B.862.995.985.8681.000n1000 xyy
3Qwen3-VL-30B-A3B.861.995.984.8671.000n1000 xyy
4InternVL3-38B.744.941.832.6750.999n1000 xyy
5LLaVA-NeXT-Vicuna.701.977.885.4441.000n01 xyy
6Gemma-4-12B.589.749.631.4750.999n1000 xywh
–mean gold box.415.784.369.025
7InternVL3.5-38B.412.605.329.1020.997abs xywh
8Gemma-3-12B.329.525.263.0541.000abs xywh
9Qwen2.5-VL-7B.308.529.210.0030.998abs xywh
10Qwen2.5-VL-3B.307.527.210.0031.000abs xywh
11InternVL3.5-30B.289.446.174.0530.995abs xywh
12InternVL3-8B.246.399.200.0860.940abs xywh
13InternVL3.5-8B.236.376.156.0320.965abs xywh
14Pixtral-12B.229.331.202.0770.999abs xywh
15LLaVA-NeXT-Mistral.089.239.171.1410.523abs xywh
16Llama-3.2-Vision.061.017.001.0010.996abs xywh
17Molmo2-8B.055.070.027.0180.998abs xywh
18Phi-3.5-Vision.030.026.008.0011.000abs xywh

caption: Localization. The headline column charges unparsed replies as zero. Twelve of eighteen models fall below the mean gold box, which is a rule that draws the same box on every image. The convention column records which of eight box formats the model emitted, detected by scoring under all eight. Molmo2-8B and Phi-3.5-Vision mostly repeat the example box given in the prompt.

:::

scores 0.147 under absolute xywhxywh and 0.829 under normalized xyxyxyxy in [0, 1000]. Scoring it under the wrong convention would understate it by 0.68 mean IoU and would read as a capability claim and not a parsing error. LLaVA-NeXT-Mistral parses only 52.3% of the time. Its mean IoU over parsed replies is .170 and its headline is .089. We report both so the failure is attributable to output format as well.

Relative counting

Table 44 gives the full leaderboard.

#ModelliftbalAccaccposBias
1Qwen3-VL-32B+.403.653.672.445
2Gemma-4-12B+.388.638.658.479
3InternVL3-38B+.366.616.634.513
4InternVL3.5-38B+.338.588.599.409
5Molmo2-8B+.329.579.594.235
6Qwen3-VL-8B+.320.570.587.239
7Qwen3-VL-30B-A3B+.312.562.576.394
8InternVL3.5-30B+.305.555.569.416
9InternVL3.5-8B+.277.527.540.275
10Qwen2.5-VL-7B+.273.523.536.283
11Pixtral-12B+.233.483.494.205
12Gemma-3-12B+.228.478.486.345
13Qwen2.5-VL-3B+.184.434.441.193
14Phi-3.5-Vision+.140.390.397.232
15InternVL3-8B+.098.348.357.289
16Llama-3.2-Vision+.056.306.311.262
17LLaVA-NeXT-Mistral+.005.255.256.257
18LLaVA-NeXT-Vicuna−.005.245.245.249

Table 44. Relative counting. Lift is accuracy above the .250 chance floor, normalized. The bottom model sits marginally below chance and answers one option 100% of the time.

Relative position

Table 45 gives the full leaderboard with the angular error beside the accuracy.

#ModelaccliftangErrwithin45
1Qwen3-VL-32B.550+.42527.0.895
2Gemma-4-12B.495+.37035.4.815
3Molmo2-8B.457+.33232.9.864
4Qwen3-VL-8B.421+.29640.1.803
5Qwen3-VL-30B-A3B.355+.23055.0.673
6Pixtral-12B.266+.14164.4.576
7InternVL3.5-30B.252+.12769.8.561
8InternVL3.5-8B.224+.09971.5.541
9InternVL3.5-38B.219+.09473.1.520
10Qwen2.5-VL-7B.213+.08869.2.545
11InternVL3-38B.205+.08080.5.471
12InternVL3-8B.203+.07872.9.515
13Phi-3.5-Vision.160+.03580.7.440
14Gemma-3-12B.145+.02086.9.395
15LLaVA-NeXT-Mistral.131+.00689.7.395
16Llama-3.2-Vision.1250.00089.7.382
17LLaVA-NeXT-Vicuna.1250.00090.0.375

Table 45. Relative position over eight directions on the beetle trays. The majority-direction floor is .125. Angular error is the mean error on the 45-degree ring and within45 is the share within one step. Two models sit exactly at the floor and answer one direction every time.

The OCR negative control

Locating a specimen label is a spatial act, but the answer depends on transcribing characters. There is no position, extent, direction or magnitude in the answer. We therefore run OCR on the same imagery, the same models and the same pipeline, and we hold it as a negative control of the C3 composite. The control answers the obvious objection to any coherence result. If the seven spatial tasks agree merely because larger models are better at everything, then any eighth vision task on the same imagery should agree too. Table 46 gives the four statistics with and without the control. The last row is the argument. The seven spatial tasks track parameter count at +0.588 and the control tracks it at +0.028. The coherence is therefore not a general capability gradient.

StatisticSeven spatial+ OCR
Mean inter-task ρ\rho+0.703+0.619
Krippendorff α\alpha0.9400.921
Variance on PC175%68%
Mean ρ\rho against log params+0.588+0.028

Table 46. OCR acts as the negative control. It correlates +0.366 with the seven spatial tasks, against +0.703 among themselves. Every spatial task individually beats its correlation with the control.

The constant-string exploit. OCR also surfaces a failure mode that only a floor makes visible. Specimen codes share a fixed prefix, so a model that answers the prefix and guesses the suffix scores 0.675 on whole-code character accuracy while reading nothing. We therefore report suffix character accuracy against a constant-string floor of 0.349. Table 47 gives the two columns for the four most inflated and the two least inflated models.

ModelcharAccsuffixCharAccinflation
Phi-3.5-Vision.568.161.407
Pixtral-12B.593.199.394
LLaVA-NeXT-Vicuna.607.214.393
InternVL3-38B.620.246.374
Qwen2.5-VL-3B.969.940.029
Qwen3-VL-30B-A3B.993.987.007

Table 47. Whole-code character accuracy against suffix character accuracy. The inflation column is what the common prefix boosts a model. Phi-3.5-Vision reads .161 of the informative characters against a floor of .349.

LLaVA-NeXT-Vicuna is a clear case. It gives a well-formatted answer 100% of the time and scores 0.00 EM, 0.214 suffix character accuracy, and between 0.990–1.000 per-position accuracy on the first seven characters. The format compliance, per-position profile and floor together establish that nothing was read.

Internal coherence

Table 48 gives the four coherence statistics.

StatisticValue
Mean inter-task ρ\rho+0.703
Range+0.351 to +0.909
Krippendorff α\alpha0.940
Variance on PC175%

Table 48. The seven spatial tasks cohere. The tightest pairs are viewpoint with relative counting at +0.909 and viewpoint with relative position at +0.887, which are both spatial-relation judgments.

Within the cross-component analysis, relative position at .168 and localization at .176 load lowest on the shared factor, which is consistent with them measuring something the rest of the benchmark does not.

C4 Full Results

The evaluation encompasses 18 models across 5 species and 3 conditions, resulting in 270 distinct subgroups and 270 000 individual answers. We establish verdicts based on the random condition; since each-condition pairs are separable by appearance alone, benchmarking against them would introduce circularity.

Figure 12. The seven spatial tasks track parameter count and the control does not, on the same imagery with the same models. The coherence is spatial rather than a general capability gradient.

Figure 13. Several tasks sit below what a rule that never looks at the image achieves. A benchmark reporting only accuracy would record these as partial competence.

Parsing integrity

All 270 000 answers resolve successfully. Of these, 269 903 resolve at the strictest tier (a bare Yes or No after stripping markdown, quotes, and trailing punctuation). The remaining 97 answers resolve one tier down, containing exactly one of the two target words as an isolated token. No answers require the prefix, negated, or unparsed fallback tiers. We implement two critical parser safeguards. First, the string Yes or No occurs when a model echoes the prompt. Without an explicit rule, this would incorrectly resolve to Yes via prefix matching. Second, the string I am not sure contains the word not and would resolve to No under a contains-unique rule, mistakenly converting an abstention into a confident negative answer. To prevent this, we explicitly match refusal phrasings first and route them to the unparsed tier. Consequently, no result in this evaluation depends on parser leniency.

Discriminability

Table 49 presents d′d' for every model across all three conditions. We highlight two principal findings.

Modeleasyrandomhardverdict
Qwen3-VL-32B1.4550.696−0.604weak
InternVL3-8B2.0620.684−0.738weak
Qwen2.5-VL-3B2.2900.655−0.764weak
InternVL3.5-30B1.5630.568−0.529weak
Phi-3.5-Vision1.9190.562−0.932weak
Qwen3-VL-30B-A3B1.2810.518−0.248weak
Qwen3-VL-8B1.6350.492−0.589weak
Molmo2-8B1.8200.491−0.411weak
LLaVA-NeXT-Mistral1.7380.476−0.550weak
InternVL3-38B1.5970.463−0.725weak
Qwen2.5-VL-7B1.3350.442−0.395weak
Gemma-4-12B1.3040.351−0.414weak
InternVL3.5-38B1.2780.258−0.460weak
InternVL3.5-8B0.9050.236−0.286weak
Gemma-3-12B0.9460.120−0.091degenerate
Pixtral-12B0.6510.119−0.240at chance
Llama-3.2-Vision0.8640.000−0.208degenerate
LLaVA-NeXT-Vicuna0.0000.0000.000degenerate

Table 49. Discriminability d′d' pooled across the five species. The highest random-condition d′d' in the field is 0.696, which is approximately 64% balanced accuracy. No model reaches d′=1.0d' = 1.0. Seventeen of eighteen models exhibit negative d′d' under adversarial pairing.

A performance ceiling. Individual re-identification remains largely unsolved for this generation of vision-language models, regardless of dataset manipulation. The finest granularity of visual discrimination in Bio-VQA represents the domain where current models struggle most.

Below-chance performance under adversarial pairing. Seventeen of the eighteen models exhibit negative d′d' in the hard condition, ranging from −0.932 to −0.091. The single exception produces a constant output and is immune to dataset manipulation.

Model scale does not reliably predict performance; for instance, Qwen2.5-VL-3B ranks third, outperforming the significantly larger InternVL3-38B.

Response bias

Table 50 presents the criterion cc for the seven most biased models.

Modelcc easycc randomcc hard
LLaVA-NeXT-Vicuna−3.09−3.09−3.09
LLaVA-NeXT-Mistral−2.41−2.46−2.54
Gemma-3-12B−1.60−1.99−1.99
Llama-3.2-Vision−1.13−1.59−1.59
Qwen2.5-VL-7B−1.18−1.34−1.51
Qwen2.5-VL-3B−0.88−1.32−1.53
Qwen3-VL-30B-A3B+0.13+0.15−0.06

Table 50. Criterion cc, pooled. Negative values indicate a standing preference for Yes. 17 of the 18 models lean toward Yes, and this bias deepens from easy to hard in nearly every case. Six models exceed ∣c∣=1.0|c| = 1.0 in the neutral condition, for which we report d′d' and balanced accuracy rather than raw accuracy.

As image pairs become increasingly confusable, models consistently become more affirmative rather than exhibiting appropriate caution. This behavior contradicts expectations for a model actively tracking its own uncertainty. Because this trend holds across different model families and scales, it reflects a general vulnerability rather than an architecture-specific artifact.

The difficulty manipulation

To validate our difficulty manipulation, we quantify its impact empirically. Table 51 details the easy-minus-hard accuracy difference for every model, alongside 95% confidence intervals.

Modeleasy − hard95% CI
InternVL3-8B+38.8[+37.0, +40.6]
InternVL3.5-30B+38.4[+36.6, +40.1]
InternVL3-38B+38.4[+36.6, +40.2]
Phi-3.5-Vision+37.0[+35.2, +38.8]
Qwen3-VL-32B+36.9[+35.1, +38.7]
Molmo2-8B+36.0[+34.2, +37.7]
Qwen2.5-VL-3B+35.3[+33.5, +37.1]
LLaVA-NeXT-Mistral+5.2[+3.2, +7.1]
LLaVA-NeXT-Vicuna+0.0[−2.0, +2.0]

Table 51. Accuracy difference in percentage points between the easy and hard conditions, pooled, with two-proportion difference intervals. Seventeen models confirm the effect, one remains flat, and none reverse.

Analyzed by species, 77 out of 90 model-species combinations exhibit a weakly monotonic easy ≥\geq random ≥\geq hard ordering in balanced accuracy. Under a null hypothesis where the condition is irrelevant, a random ordering of three conditions would be monotonic with a probability of 1/6. The exact binomial probability of observing 77 or more monotonic orderings out of 90 is 1.9×10−461.9 \times10^{-46}. The single flat case corresponds to the model generating a constant answer, which by construction cannot react to visual manipulations.

By species

Table 52 provides the mean d′d' separated by species. We anticipated the anomalous beluga result based on dataset statistics prior to model evaluation. Beluga is the only species for which the random condition yields a negative similarity margin (−0.0613). Consequently, under uniform sampling, different-individual pairs are on average more visually similar than same-individual pairs. Because the DINOv2 embedding geometry for belugas is poorly organized around individual identity, manipulations defined within this space are inherently less effective.

Specieseasyrandomhardeasy − hard
Whale shark2.8120.276−0.9063.718
Spotted hyena1.7430.672−0.1261.869
Grevy’s zebra1.3460.391−0.5961.942
Leopard1.2800.517−0.1881.468
Beluga whale0.5080.383−0.7581.267

Table 52. Mean d′d' across models, partitioned by species. Beluga is diagnostically useful, as its easy-condition separation is notably the weakest.

The specialist baseline

Table 53 demonstrates the performance of the specialist model across all three conditions.

ConditionAUCd′d'accuracy
easy0.9272.0550.902
random0.6980.7330.683
hard0.247−0.9690.317

Table 53. MegaDescriptor evaluation on Grevy’s zebra. The threshold is fit on the random condition and applied unchanged to both easy and hard conditions, making the easy and hard rows out-of-fit evaluations.

The specialist model performs well on the easy and random conditions, but falls below chance on the hard condition. It demonstrates an effect magnitude that exceeds generalist VLMs. Because this system relies strictly on thresholded cosine similarity and lacks any language model or reasoning component, its performance drop is driven exclusively by the appearance anti-correlation induced by the adversarial condition. This provides compelling evidence that the adversarial condition successfully isolates and targets appearance-based matching rather than inadvertently penalizing linguistic reasoning. We do not claim a fundamental capability gap between VLMs and the specialist. On neutral pairs, the specialist achieves a d′d' of 0.733 compared to the best VLM’s 0.696, effectively a statistical tie. These pairs represent a formidable challenge for appearance-based methods in general.

Construction check

Using the same evaluation script, we score the raw DINOv2 similarities stored in the pair metadata. While these scores do not constitute a baseline, we report them to transparently quantify the degree of construction circularity. Table 54 shows the AUC of the embedding itself in each condition.

Specieseasy AUCrandom AUChard AUC
Leopard1.0000.8350.114
Whale shark1.0000.7180.053
Spotted hyena1.0000.7840.095
Grevy’s zebra1.0000.6580.068
Beluga whale0.9900.3860.010

Table 54. The DINOv2 similarity itself, scored on the pairs it defines. An easy-condition AUC of 1.000 confirms the construction operates exactly as intended. We report this to illustrate the empirical upper bound of the circularity concern.

Why below chance

A natural objection is that 0.500 should serve as a performance floor for a binary task, suggesting that below-chance accuracy implies the model knows the correct answer but deliberately reports the opposite. We address this directly. An accuracy of 0.500 is the expected value for a responder whose output is entirely independent of the stimulus (i.e. random guessing). For any system whose output is conditioned on the input image, performance can be driven below 0.500 through adversarial sample selection, which requires no latent knowledge of the true label.

The mechanism. Models implicitly apply a decision rule that approximates answer yes if the two animals look alike. Because this rule generally correlates with ground truth, these models score above chance in the random condition. However, the hard condition explicitly selects pairs where visual similarity is anti-correlated with true identity. The models apply their appearance-based rule unchanged, but we have adversarially arranged the sample such that this otherwise sound rule consistently points to the incorrect answer.

Modeld′d^\prime randomd′d^\prime hard∣d′∣|d^\prime|
InternVL3-8B0.684−0.738-0.7380.68 to 0.74
Qwen3-VL-32B0.696−0.604-0.6040.70 to 0.60
Qwen2.5-VL-3B0.655−0.764-0.7640.66 to 0.76
Molmo2-8B0.491−0.411-0.4110.49 to 0.41

Table 55. The magnitude of the sensitivity survives the manipulation; only the sign changes. This rules out the hypothesis that the model lacks signal and is merely guessing, which would drive d′d^\prime toward 0 rather than to −0.7-0.7.

What negative d′d^\prime does and does not establish. A negative d′d^\prime establishes that the model’s decision variable is anti-correlated with the ground truth label within this specific sample. It does not establish that the model’s internal decision variable represents actual identity. We posit that this variable is simply appearance similarity, a proxy that normally correlates with identity, but is intentionally decorrelated from it in this condition. Under this interpretation, the model never successfully forms an abstract identity judgment, meaning there is no true identity judgment being inverted. Two evidences support this mechanism. First, a specialist system incapable of deliberation (MegaDescriptor) exhibits the same vulnerability, reaching d′=−0.969d^\prime= -0.969 in the hard condition. A thresholded dot product cannot implicitly know an answer and willfully choose the opposite. Second, the magnitude of the model’s sensitivity is largely preserved while only the sign flips. Tab. 55 pairs the random and hard columns for four models to illustrate this. This behavior is the hallmark signature of a constant heuristic mechanism being evaluated against inverted labels.

Figure 14. Difficulty is set by DINOv2 embedding similarity. In the hard condition appearance and identity disagree, so a model reading resemblance rather than identity scores below chance.

Why retrieval is viewpoint-constrained

We constrain our retrieval process to within specific viewpoint groups rather than searching across the entire catalog. Because viewpoint dominates the DINOv2 embedding space, not having this constraint would mean an anchor’s most dissimilar neighbors would almost universally feature the opposite viewing angle. Consequently, hard same-individual pairs would merely consist of a left-flank photograph paired with a right-flank photograph, degenerating the evaluation from true identity discrimination into trivial viewpoint matching. Constraining retrieval strictly to a viewpoint group forces the embedding to discriminate based on features other than pose, ensuring the hard condition effectively evaluates fine-grained identity matching as intended.

Figure 15. Seventeen of eighteen models and the specialist all fall below chance under adversarial pairing, while preserving the magnitude of their sensitivity.

Metric Validation

We check CAGS two ways, and we audit the other three components.

The cags sensitivity sweep

We re-score every model under a grid of configurations on a fixed, family-stratified 60 000-row subsample per model, with identical rows across configurations. Let ri(0)r_i^{(0)} and ri(c)r_i^{(c)} be model ii’s rank under the default and under configuration cc. Then

ρ=1−6∑idi2n(n2−1),di=ri(0)−ri(c)(33)\rho= 1 - \frac{6\sum_i d_i^2}{n(n^2-1)}, \qquad d_i = r_i^{(0)} - r_i^{(c)} \tag*{(33)}

and displacement is max⁡i∣di∣\max_i |d_i|. Table 56 gives ρ\rho, Kendall τ\tau and displacement for every configuration.

Configρ\rhoτ\taudisp.top-3mean ∣Δ∣|\Delta|
default1.00001.00000✓0.0000
τ=20\tau= 200.99790.98691✓0.0008
τ=40\tau= 400.99790.98691✓0.0008
τ=50\tau= 500.99790.98691✓0.0015
α=0.3\alpha= 0.31.00001.00000✓0.0011
α=0.4\alpha= 0.41.00001.00000✓0.0006
α=0.6\alpha= 0.61.00001.00000✓0.0007
α=0.7\alpha= 0.70.99790.98691✓0.0015
linear kernel0.99590.97391✓0.0027
prefer text0.99790.98691✓0.0004
anat in vis0.95870.85623✓0.0200

Table 56. The CAGS sensitivity sweep across eleven configurations. The base grid varies the color-kernel temperature τ\tau, the region-value weight α\alpha, and the color kernel shape. The policy grid adds the letter against text resolution preference and one substantive modeling toggle.

Across the base grid, the minimum ρ\rho is 0.9959, the minimum τ\tau is 0.9739, the maximum displacement is one rank, and the top three set is stable in every configuration. The only rank exchanges are single-position swaps between models whose bootstrap intervals already overlap, which are statistical ties rather than instability. The one configuration that moves mid-table ranks is folding anatomy part similarity into the visibility set kernel, so that naming the left wing when the right wing was visible earns partial credit. This is a substantive modeling decision and not an arbitrary constant, and it changes the visibility-family scores. Table 57 names the models that move under it. We adopt the EM default and we disclose that the alternative reshuffles the middle of the table by up to three positions while leaving the podium intact. It shows that the ranking’s one sensitivity is a known semantic knob and not a fragile numeric one.

Modeldefaultanat in visΔ\Delta
Gemma-3-12B129+3
Llama-3.2-Vision1715+2
Qwen2.5-VL-7B810−2
InternVL3.5-8B1113−2
LLaVA-NeXT-Vicuna1518−3
Top three1, 2, 31, 2, 30

Table 57. Models movement under one substantive toggle. Top three set is unchanged and the movement is confined to families where crediting near-miss parts is a defensible design choice.

The human study

CAGS could in principle produce the model ordering by itself, so we compare it against human judgment on a blind stratified sample.

Sample design. We draw 306 items across a category by CAGS-band grid. The three bands are low with CAGS in [0, 0.2), mid with CAGS in [0.4, 0.6), and high with CAGS in [0.8, 1.0]. Each band contributes 34 items per category and 102 items overall. The five families represented are attr_bare_value with 102 items, attr_group_value with 102, visibility_side with 41, visibility_partset with 40 and marker_localization with 21.

Protocol. Each item renders as a card showing the image, the question, the reference answer and the model answer. Model identity, the CAGS value and the band are hidden. Three annotators rate each card on a three-level ordinal scale. A rating of 2 is correct or equivalent to the reference. A rating of 1 is incorrect but close, which covers the right body region with the wrong property, an adjacent color, an off-by-one size, one of two correct parts, or a near-miss location. A rating of 0 is unrelated or far off. An empty cell is an abstention. We treat it as a missing value, not a zero. We aggregate the ratings into hi∈[0,1]h_i \in[0,1], which is the weight-renormalized mean over the annotators who rated item ii, rescaled to the unit interval. Table 58 gives the reliability statistics.

StatisticValuenn
Krippendorff α\alpha, ordinal0.7937904
Mean pairwise quadratic κ\kappa0.7914
κ\kappa, annotator 1 against 20.7579292
κ\kappa, annotator 1 against 30.8312306
κ\kappa, annotator 2 against 30.7851292
Abstention, any annotator0.0458306
Abstention, all annotators0.0000306

Table 58. Inter-annotator reliability. Every item receives at least one rating.

Table 59 gives agreement between the metric and the annotators.

StatisticAll, n=306n = 306Complete, n=292n = 292
τb\tau_b, CAGS against human0.58830.5879
95% interval[0.515, 0.657][0.514, 0.659]
τb\tau_b, EM against human0.71780.7245
Difference, CAGS minus EM−0.1295−0.1366
ρ\rho, CAGS against human0.73280.7330
ρ\rho, EM against human0.80440.8102
Quadratic κ\kappa, metric against human0.7187
Mean pairwise annotator κ\kappa0.7914
Ceiling ratio0.908

Table 59. Agreement between the metric and the annotators. The ceiling ratio is the metric κ\kappa against human κ\kappa divided by the mean pairwise annotator κ\kappa. CAGS reaches 0.908 of the rate at which the annotators agree with each other.

Where CAGS correlates with human. EM agrees with the annotators more closely than CAGS does on the full sample, which is expected. The band construction makes EM a deterministic function of the band, since every low and mid item has EM = 0 and every high item has EM = 1. So the interesting question is what happens inside the EM-zero subset, where EM carries no information at all.

Tab. 60 gives the separation CAGS achieves there.

Tab. 61 gives the mean metric and human score in each band.

Where CAGS disagrees with humans. We report disagreements in Tab. 62, family by family.

The mechanism is the laterality confusion we describe in Sec. F. The three worst-disagreeing items are all left wing answered as right wing or the reverse, where CAGS scores between 0.559 and 0.588 and all three annotators score zero.

StatisticValueCounts
Items with EM = 0204102 low, 102 mid
AUC, credited against rejected0.6755[0.591, 0.757]
AUC, partial against rejected0.6823strict variant

tbl:60. Within the 204 items EM scores zero, CAGS separates the answers annotators credited from the ones they rejected. EM cannot, because it assigns all 204 the same score.

Bandnnmean CAGSmean hh
low1020.03570.2484
mid1020.46470.3595
high1021.00000.9861

tbl:61. The human label tracks the band monotonically. The mid band is where the two diverge most, which is where partial credit is being tested.

Familynnτb\tau_b CAGSmean CAGSmean hh
attr_bare_value1020.66920.49990.5596
attr_group_value1020.55440.51300.5441
visibility_side410.56380.54150.6301
visibility_partset400.80630.36110.4542
marker_localization210.44310.62310.2857

tbl:62. Agreement by family. marker_localization is the weakest agreement and the largest gap between metric and human, with mean CAGS at 0.6231 against mean human at 0.2857.

ReferenceResponseCAGShh
(C) left wing(D) right wing0.58840.0
(A) left wing(C) right wing0.56930.0
(D) left wing(C) right wing0.55940.0

tbl:63. The three items where CAGS and the annotators disagree most. The anatomy kernel places the left and right members of a paired part close together, so it over-credits exactly the confusion that matters. We state this as a limitation in the main paper and we compute the laterality analysis on EM for that reason.

Tab. 63 gives the three items.

The disagreement runs the other way as well. On four items the annotators credit a response that CAGS scores at exactly zero, all of them cases where the reference is None and the model names a part. Annotators judged the part visible in the image and the reference wrong. These are annotation errors in the source, not metric errors.

Leave-one-annotator-out. We check that no single annotator drives the result. Table 64 repeats the three headline statistics with each annotator dropped in turn.

Droppedτb\tau_b CAGSAUCκ\kappa metric-human
none0.58830.67550.7187
annotator 10.53060.60710.6464
annotator 20.62940.70630.7189
annotator 30.60390.68050.6782

Table 64. Leave-one-annotator-out. The conclusion holds in every subset. The unique-dissent rates are 0.103, 0.182 and 0.055 respectively.

The C1 synonym audit. Every soft credit is re-verified against GBIF by a separate script that reads the soft column the scorer wrote and cannot conjure credits the scorer did not compute. In the audited run, 100% of the roughly 130 000 soft credits resolve to a valid old-to-new substitution and zero are flagged for review. The order build, run, check is a data dependency. The build stage writes the synonym map. The run stage is the only stage that computes the soft column. The check stage reads what the run stage wrote.

A bird image, question, answer options, and model response

Figure 16. The annotators see the image, the question, the reference and the response. They do not see the model identity, the metric value or the stratum.

C3 floors and shortcut controls

Each C3 task carries at least one control that closes a specific route to a high score without reading the image. Absolute distance carries three, described in Sec. C. Counting carries the rank correlation, which is exactly zero for any constant predictor. Viewpoint carries three floors that differ by a factor of three, so a score quoted without its floor is not interpretable. Localization carries the mean gold box. OCR carries the constant string. We fit every floor on the same row subset as the model under test, so a model with a lower parse rate is compared against a baseline recomputed on its own rows.

Reading CAGS and GSraw\mathrm{GS}_{\mathrm{raw}} together

We report the mean raw score alongside CAGS. The gap GSraw−CAGS‾\mathrm{GS}_{\mathrm{raw}}-\overline{\mathrm{CAGS}} is large exactly when a family’s questions are easy to look good on, which happens when the distractors sit close to the ground truth. That gap is a diagnostic of question design and not of the model, and we report it so a reader can tell the two apart.

Figure 17. Inside the EM-zero region, cags separates credited from rejected answers. The three circled points are the laterality items, which are the metric’s one systematic disagreement with the annotators.

ComponentInternal ρ\rhoKrippendorff α\alpha
C1.841.981
C3.662.925
C2.645.901
C4.617.889

Table 65. Within-component agreement. Tasks inside a component agree enough for the component score to summarize them, and differ enough to be worth reporting on their own.

Cross-Component Analysis

Construction

We build an 18 × 24 matrix of model by task scores, using each task’s headline metric. We correlate every pair of tasks with Spearman ρ\rho over the eighteen models. We then summarize within-component and across-component agreement, and we run a principal component analysis on the standardized matrix.

Correlating tasks over 18 models is a small-nn analysis. The intervals are wide. They are wide enough to separate correlated pairs from uncorrelated ones and not wide enough to rank one pair above another. We treat the block structure as the finding and individual pairwise values as descriptive.

Tasks agree more inside a component than across one

Task pairs within a component correlate at mean ρ=.749\rho= .749. Pairs drawn from different components correlate at .534. A label permutation test puts p=.0005p = .0005 on that gap.

Tab. 65 gives internal agreement per component.

C1 is the most internally coherent component, which is consistent with the 10 × 10 group matrix in Tab. 24. C4 is the least, which is consistent with it being the component that dissociates.

The component correlation matrix

Tab. 66 gives the four-by-four matrix.

Spearman ρ\rhoC1C2C3C4
C1—.689.787.433†
C2—.927.514†
C3—.610
C4—

Table 66. Component-level correlation over the 18 models. † not significant after Holm correction. The mean across components is .660 against .691 within them.

StatisticValueNote
Median model rank movement5.5 of 18across components
Largest movement11 of 18Molmo2-8B
Molmo2-8B C1 percentile3rd
Molmo2-8B C2 percentile79th
Phi-3.5 C1 percentile8th
Phi-3.5 C4 percentile64th

Table 67. Rank movement between components. Three models appear in every component’s top five. Below them the ranking reorders substantially, so anyone choosing a mid-field model cannot do it from one number.

ComponentLowest-loading taskLoading
C4all five species.091 to .185
C2laterality.154
C3relative position.168
C3localization.176
C1gymnosperms.183

Table 68. The tasks that load lowest on the shared factor. Each is a task we designed to isolate something the rest of the benchmark does not measure, and each sits where we designed it to sit.

What the components buy is everything below the top

Tab. 67 shows the movement statistics. Molmo2-8B moves eleven of eighteen places, which is the largest movement in the benchmark. It is the only model trained with explicit pointing supervision, and that supervision buys grounding without buying taxonomy.

Where the dissociations sit

Tab. 68 reports the lowest-loading task in each component. We report this as a prediction that held and not as a post-hoc reading. The adversarial C4 condition, the laterality gate, the relative-position isolation and the OCR control were all specified before we saw any correlation.

Imputation sensitivity

Not every model has a score on every task. Two InternVL runs were superseded and we exclude them from C4 rather than deleting their files. Where a cell is missing we impute the component median and we re-run the analysis with the affected model dropped entirely. The block structure is unchanged under both treatments and the mean within-against-across gap moves by less than .01.

Figure 18. The effective structure is three blocks rather than four. C2 and C3 merge at the reliability we can measure. C4 separates.

Prompts and Parsing

Every model receives the same question and the same answer-format instruction within a question family. We list both here exactly as they appear in the released question files and inference scripts. The instruction is appended to the question in the user turn. Three model-specific wrappers apply on top of these strings: Gemma-3 and Gemma-4 add the system message You are a precise taxonomy expert. Reply with only the requested term. on C1 and C2; InternVL3-8B, InternVL3-38B and InternVL3.5-38B prefix Please answer in English.; InternVL uses its checkpoint’s default system message, and the other models use none. C4 uses no system message for any model. All runs use greedy decoding.

C1 prompts

Each item draws one question template at random (seed 42), with {rank} set to the asked rank (species epithet for the species rank) and {label} set to animal, plant or organism by kingdom. Question templates (phylum to species epithet).

What is the {rank} of the {label} shown in this image?

Which {rank} does the {label} in this image belong to? Identify the {rank} of the {label} shown in this image. Name the {rank} of the {label} in this image. What is the correct {rank} label for the {label} in this image?

Figure 19. All tasks load positively on the first component, so a single ability drives most of the variance. The tasks with the lowest loadings are exactly the ones we designed to dissociate.

Figure 20. Three models stay in the top five everywhere. The median model moves 5.5 of 18 places between components, so one leaderboard answers which model is best and not which model is best for a given task.

Question templates (scientific name and common name).

What is the {rank} of the {label} shown in this image? Identify the {rank} of the {label} shown in this image. Name the {rank} of the {label} shown in this image.

Multiple-choice common name. The four options are shuffled and listed without letters.

Among the four options, what is the common name of the {label} shown in this image? {o1}, {o2}, {o3}, {o4}

Answer-format instruction, appended by rank.

phylum: Answer with ONLY the Phylum name (e.g., Arthropoda). Do not include class, order, or any other taxonomic ranks. Do not write full sentences.

class: Answer with ONLY the Class name (e.g., Arachnida, Insecta). Do not include order, family, or any other taxonomic ranks. Do not write full sentences.

order: Answer with ONLY the Order name (e.g., Araneae). Do not include family, genus, or any other taxonomic ranks. Do not write full sentences.

family: Answer with ONLY the Family name (e.g., Araneidae). Do not include genus, species, or any other taxonomic ranks. Do not write full sentences.

genus: Answer with ONLY the Genus name (capitalized, e.g., Argiope). Do not include the species epithet or family name. Do not write full sentences.

species epithet: Answer with ONLY the species epithet (the single, lowercase second word of the scientific name, e.g., aurantia). Absolutely DO NOT include the capitalized Genus name. Do not write full sentences.

scientific name: Answer with ONLY the full binomial scientific name (Genus and species, e.g., Argiope aurantia). Do not include the common name or higher taxonomic ranks. Do not write full sentences.

multiple choice: Answer with ONLY the exact correct option from the four choices provided in the question. Absolutely DO NOT write full sentences, and do not provide any extra context or scientific names.

common name: Answer with ONLY the everyday English common name (e.g., Orb weavers). Absolutely DO NOT provide the Latin scientific name. Do not write full sentences.

Generation is capped at 15 new tokens. The inference scripts keep only the last word of a multi-word species-epithet reply, which understates that rank; we analyze species-level naming through the scientific name.

C2 prompts

Table 69 lists the ten C2 question templates. Every multiple-choice item then receives the instruction below; open items receive the instruction for their family.

Multiple-choice instruction.

[CRITICAL INSTRUCTIONS – NO VIOLATIONS ALLOWED]: 1. Locate the ‘Options:’ section in the question above. 2. Identify the single correct choice from that list. 3. Output ONLY the matching option

FamilyTemplate
attr_bare_valueWhat is the {part} {property} of this bird? Options: (A) . . .
attr_group_valueWhich of the following is the most prominently visible physical trait of the bird in this image? Options: . . .
marker_localizationWhat is the body part of this bird marked with the {color} {shape}? Options: . . .
visibility_partsetWhich of the following parts of the bird are visible in this image? Options: (A) {x}, (B) {y}, (C) {z}, (D) None, (E) All three
visibility_sideWhich of the {eye—leg—wing}s of the bird are visible in this image? Options: (A) Left, (B) Right, (C) Both, (D) None
category_mcqWhat type of bird is this in the image? Options: . . .
open_common_nameWhat is the common name of the bird?
open_scientific_nameWhat is the scientific name of the bird?
open_genusWhat is the genus of the bird?
open_kind_ofWhat kind of {category} is the bird in this image?

caption: tbl:69. The ten C2 question templates. The marker family specifies the marker using both color and shape, as neither cue alone is sufficient for identification.

:::

letter in parentheses and its exact trailing text string. For example, if the correct option is ’(X) alpha bravo charlie’, your entire response must be exactly: (X) alpha bravo charlie

  1. DO NOT truncate the response to just the option letter (e.g., do not output just ’(X)’).

  1. DO NOT invent, hallucinate, or insert any placeholder words from external examples. Absolutely DO NOT output any word unless it is explicitly written inside the chosen option string above.

  1. Output exactly zero extra words, zero markdown formatting, and zero punctuation outside the option string.

Open-ended instructions.

scientific name: Answer with ONLY the scientific name of the bird. Do not write full sentences.

genus: Answer with ONLY the capitalized Genus name of the bird. Do not write full sentences.

common name, kind of: Answer with ONLY the everyday English common name of the bird. Do not write full sentences.

other: Answer directly and concisely with only the exact terms requested. Do not write full sentences.

C3 prompts

Each C3 prompt is a frame describing the image, the question, and an answer-format instruction, in that order. tbl:70 gives the question for each task; the frames and instructions follow.

Frames.

TaskQuestion
CountingHow many {animals} are there in this image? (two-panel tiers: . . . in the left/right photograph?)
Relative countingWhich of these 4 images contains {most—least} number of {animals}: Image A (top-left), Image B (top-right), Image C (bottom-left), Image D (bottom-right)?
Absolute distanceWhat is the measurement of the elytra {length—width} (marked with red line with white border) with respect to the scalebar (marked with blue line with green border) in cm?
Relative distanceAmong the four highlighted beetle specimens, which one has the largest elytral {length—width}? Beetle A (red dotted box/line), Beetle B (blue dotted box/line), Beetle C (green dotted box/line), and Beetle D (white dotted box/line)
LocalizationProvide bounding box coordinates for the subject animal ({species}) in this image.
Relative positionThis image is marked with six boxes: two red, two blue, one green and one cyan. Exactly one red box and exactly one blue box contain a specimen; the other red box and the other blue box are empty. The green and cyan boxes are irrelevant to the question. Where is the specimen inside the red box located with respect to the specimen inside the blue box? Options: {four of the eight directions}
ViewpointWhat is the viewpoint of the animal (red bounding box) shown in this image? (other tiers: What is the viewpoint of the {species} in this image? / . . . marked in the {left—middle—right} bounding box?)
OCRWhat is the text under the barcode (starting with A0...) shown in this image?

caption: tbl:70. The eight C3 questions. Each names the target, so a non-localization question does not require the model to find the subject first.

:::

counting, two-panel tiers: This image consists of two separate photographs placed side by side. Count only within the photograph named in the question and ignore the other one.

relative counting: This image is a 2x2 grid of four separate photographs: Image A – top-left, Image B – top-right, Image C – bottom-left, Image D – bottom-right.

absolute distance: This image shows a plate of pinned beetle specimens, photographed at a single scale. A BLUE line marks the scale bar. The scale bar is exactly 1 cm long. A RED line marks the measurement asked about below.

relative distance: This image shows a single plate of pinned beetle specimens, photographed at one magnification. Four specimens are marked, each with a dotted box and a coloured line drawn along its elytra: Beetle A – red, Beetle B – blue, Beetle C – green, Beetle D – white.

localization: The whole image is {W} pixels wide and {H} pixels tall.

viewpoint, three-panel tiers: This image consists of three separate photographs placed side by side. OCR: This image shows a specimen label with a barcode and a printed accession code beneath it.

Answer-format instructions.

counting: Count every individual, including any that are partially visible or cut off at the edge of the photograph. Give a single best estimate even if you are uncertain. Respond with one integer and nothing else: no words, no units, no ranges. If you count seven, respond: 7

relative counting: Count the animals in each of the four photographs and compare the totals. The photographs are shown at different scales, so compare how many animals there are, not how large they appear. Exactly one photograph is the answer. Choose it even if the difference is small. Answer immediately with the letter. Do not describe the photographs and do not explain your reasoning. Respond with one letter and nothing else: A, B, C or D.

absolute distance: Compare the pixel length of the red line to the pixel length of the blue scale bar. Because the scale bar is exactly 1 cm long, the red line’s length in centimetres equals that ratio: if the red line is half as long as the blue line the answer is 0.50 cm, and if it is twice as long the answer is 2.00 cm. Measure the two lines as they appear in this image. Do not answer from what you know about typical beetle sizes. Respond with one number rounded to two decimal places followed by cm, and nothing else. No explanation, no range, no other units. Example: 0.87 cm

relative distance: Compare the lengths of the four coloured lines as they appear in this image. Because all four specimens are in the same photograph at the same magnification, the line drawn on screen is directly proportional to the real measurement - no scale conversion is needed. Exactly one of the four is the largest. Choose it even if the difference is small. Respond with one letter and nothing else: A, B, C or D.

localization: Respond with four numbers in this exact format and nothing else: [x, y, width, height] where x and y are the pixel coordinates of the top-left corner of the box, measured from the top-left corner of the WHOLE image described above, and width and height are the size of the box in pixels. Do not use fractions, percentages or normalised coordinates. Do not explain. Example: [1240, 88, 540, 410]

relative position: Answer with one of the four options exactly as written above, in lower case, and nothing else. Do not describe the boxes, do not explain your reasoning, do not restate the question. For example, if the answer is the option bottom-left, reply: bottom-left

viewpoint: Answer with the direction the animal is facing, relative to the camera. Choose exactly one of these accepted answers: front, back, left, right, frontleft, frontright, backleft, backright, up, down. Reply with one of those options, written exactly as above in lower case, and nothing else. No explanation, no punctuation. For example, if the animal faces away from the camera and to the left, reply: backleft

OCR: Read the printed characters underneath the barcode. The code is always the letter A followed by eleven digits, twelve characters in total, with no spaces or punctuation. Transcribe exactly what is printed. Do not guess a plausible code and do not correct what you see. Respond with the twelve-character code and nothing else. Example: A00000012345

C4 prompt

Every pair uses the same question and instruction.

This image shows two images placed side by side. Is the main subject in the left image the very same individual animal as the main subject in the right image?

  • The image shows two photographs placed side by side.

  • Each photograph may contain more than one animal – consider only the main subject in the foreground.

  • Judge only from what is visible in the two photographs.

  • Answer with one word: Yes or No. Do not explain.

Because we rely on a single phrasing, as discussed in Sec. M, this design cannot disentangle an architectural bias toward answering Yes from artifacts introduced by the prompt’s specific wording.

The parser

The C2 scorer resolves every reply in the five steps below. The other components parse the answer forms listed in Table 14.

Step 1: Normalize. We convert text to lower-case, map underscores to spaces, standardize grey and gray, strip all punctuation (excluding colons, hyphens, and parentheses), and collapse whitespace. Consequently, Blue Jay, blue jay, and blue jay are treated as identical.

Step 2: Detect the format. The literal token Options: routes an item into the multiple-choice branch, which constructs an ordered letter-to-text mapping. All other inputs are treated as open-ended.

Step 3: Parse the gold standard. We process gold answers stored either in the (A) text format or as bare values.

Step 4: Resolve the prediction. This step is highly consequential and deliberately permissive to ensure visual competence is not penalized for formatting quirks. In order, we: strip conversational scaffolding and retain the first line; detect any explicit letter markers; identify verbatim option text; and reconcile the two. If EM fail, we fall back to a fuzzy string match with a 0.86 threshold; otherwise, we record a parsing failure and assign a score of zero. In cases where the letter and the text disagree, we record a mismatch and break the tie in favor of the letter, in accordance with our default policy.

Step 5: Route to a family. We match template phrases in priority order. The marker phrase routes to marker_localization, while the salience phrase routes to attr_group_value. The visibility phrase routes to either visibility_side or visibility_partset, depending on whether the options consist of the four side labels. Naming phrases route to the naming families, and a bare attribute question with provided options routes to attr_bare_value. Both the graded and exact-match evaluators execute Steps 1 through 5 verbatim, ensuring both metrics partition the 824 104 C2 items identically. Consequently, Table 33 and Table 34 are comparable cell-by-cell.

#Modelattr_bareattr_groupcat_mcqmarkercommongenuskind_ofscivis_setvis_side
1Qwen3-VL-32B.644.584.959.727.619.696.763.629.435.526
2Qwen3-VL-30B-A3B.631.558.951.660.608.423.733.594.434.445
3InternVL3-38B.645.562.948.603.495.636.665.614.373.484
4Gemma-4-12B.621.592.772.720.286.121.615.105.385.488
5Qwen3-VL-8B.624.531.958.608.543.486.691.553.426.435
6InternVL3.5-38B.638.560.920.656.389.417.518.454.310.498
7Molmo2-8B.643.503.532.743.284.147.579.129.366.746
8Qwen2.5-VL-7B.580.541.962.563.627.636.736.616.329.377
9InternVL3.5-30B.609.537.869.574.301.505.606.433.328.350
10InternVL3-8B.584.508.884.468.374.523.644.499.332.353
11InternVL3.5-8B.603.495.783.499.277.347.557.342.348.409
12Gemma-3-12B.551.466.915.572.438.299.688.243.416.339
13Qwen2.5-VL-3B.501.427.926.334.566.565.665.597.399.373
14Pixtral-12B.495.447.847.529.396.259.613.235.328.225
15LLaVA-NeXT-Vicuna.454.457.550.254.142.027.231.056.358.420
16Phi-3.5-Vision.534.454.716.337.177.066.096.049.108.245
17Llama-3.2-Vision.483.317.859.488.624.186.704.252.260.307
18LLaVA-NeXT-Mistral.484.372.636.392.154.023.338.063.210.377
mean.574.495.833.540.406.353.580.359.341.411

Table 33. Colors mark the best, second best, second worst and worst model in each column. CAGS by question family for all 18 models. Hardest questions are concentrated in two areas: open-ended taxonomic naming and anatomical set/side reasoning. In contrast, coarse recognition via category_mcqcategory\_mcq is nearing saturation with a field mean of .833.

#Modelattr_bareattr_groupcat_mcqmarkercommongenuskind_ofscivis_setvis_side
1Qwen3-VL-32B64.5159.0195.9061.1535.4969.4651.1551.8034.1255.65
2Qwen3-VL-30B-A3B63.1655.5395.1151.9233.5740.9644.1747.6132.0746.34
3InternVL3-38B64.4456.5494.8147.2518.9463.2234.1450.3823.0249.94
4Gemma-4-12B63.1760.4077.2059.9312.7312.5323.784.7319.3052.84
5Qwen3-VL-8B62.8153.3595.7747.1225.1347.8333.8541.5633.4046.10
6InternVL3.5-38B64.4656.4692.0054.0113.1840.8921.6834.3821.6353.08
7Molmo2-8B64.5651.0353.1865.417.8614.9715.615.5034.3771.11
8Qwen2.5-VL-7B58.2454.8696.1643.1834.0063.2846.7552.1717.0538.96
9InternVL3.5-30B60.5054.4586.8643.1611.8850.1522.4333.2823.5037.24
10InternVL3-8B58.1952.8488.4241.3513.3051.6525.1638.2322.1435.82
11InternVL3.5-8B59.2949.6478.3137.578.7534.9814.7624.0313.8543.99
12Gemma-3-12B54.7346.3791.4542.5417.2729.9226.8013.4041.4829.41
13Qwen2.5-VL-3B48.3741.8692.6428.6133.1956.5239.6748.4041.2237.29
14Pixtral-12B49.2046.0184.6837.228.9225.3236.1015.2317.9332.60
15LLaVA-NeXT-Vicuna45.3043.5655.0225.961.143.014.722.5615.4619.44
16Phi-3.5-Vision53.5446.1171.6028.303.167.261.942.8018.0426.28
17Llama-3.2-Vision46.7532.7485.8637.7234.5418.8750.7914.0029.4030.87
18LLaVA-NeXT-Mistral46.2038.8463.6435.341.532.967.032.0824.8839.08
mean57.0849.9883.2643.7617.4835.2127.8126.7925.7141.45
Familymeanminmaxreading
visibility_partset.341.108.435hardest
open_genus.353.023.696widest spread
open_scientific_name.359.049.629broadly hard
open_common_name.406.141.627free naming
visibility_side.411.225.746laterality
attr_group_value.495.317.592moderate
marker_localization.540.254.743marker binding
attr_bare_value.574.454.645most compressed
open_kind_of.580.096.763folk category
category_mcq.833.532.962near solved

Table 34. EM accuracy in percent by question family, reported on the identical item partition as Table 33. Colors mark the best, second best, second worst and worst model in each column. Coarse recognition peaks at 96.16%, whereas the highest open common-name score reaches only 35.49%. Reading the mean row against the one in Table 33 isolates where partial credit does work. The two agree to within half a point on attr_bare, attr_group, cat_mcq, genus and vis_side, and they diverge most on kind_of and common, where near misses are frequent and CAGS credits them.

Per-item outputs

We save every per-item quantity to disk. For C2, this includes the family, category, gold and predicted text, resolution status, EM, raw similarity ss, chance baseline bb, cags, and severity. For C1, we record the hard match, soft match, resolution mechanism, LCA height, and both Wu–Palmer and Lin similarities. For C3, we log the parsed prediction, gold standard, per-task metric, and the floor computed on the same rows. Finally, for C4, we store the confusion counts, parse tier, d′d', criterion cc, and the bootstrap interval. All results presented in this supplement are fully reproducible from these files.

Reproducibility

Determinism and Reproducibility

Table 71 delineates which stages of our evaluation pipeline reproduce exactly and which involve non-deterministic elements.

StageDet.Mechanism
Image download×\timesThread interleaving dictates which concurrent downloads fulfill the allocated budget. The finalized manifests serve as the artifact of record.
QA generation✓\checkmarkDeterministically seeded per component.
Manifest sharding✓\checkmarkSeed 42, applied subsequent to row identifier assignment.
Inference✓\checkmarkGreedy decoding, evaluated zero-shot with one attempt per item.
Difficulty tiers✓\checkmarkStrictly deterministic. Tie-breaking relies on taxon nomenclature.
Scoring✓\checkmarkA pure function applied to the merged prediction files. Bootstraps are fixed-seed.

Table 71. Determinism across the evaluation pipeline. Given a finalized manifest, all downstream operations are exactly reproducible. Consequently, we archive and distribute the manifests rather than expecting researchers to regenerate identical image sets byte-for-byte.

The sole non-deterministic stage is the C1 image acquisition. We query the source catalog, isolate species with complete taxonomic hierarchies, and initiate concurrent downloads constrained by a per-group image budget. The specific subset of images that fulfills this budget depends entirely on asynchronous thread interleaving. Therefore, we treat the resulting manifest as the definitive dataset artifact and release it directly.

The Verification Contract

A data row is instantiated in a C1 manifest if and only if its corresponding image is successfully written to disk and every taxonomic rank required by the generator is populated (non-null). An independent audit stage mathematically guarantees this contract holds prior to downstream execution, systematically scanning every manifest for missing, null, or empty ranks.

Bootstrap Resampling Procedure

All confidence intervals reported in this supplement are derived via percentile bootstrapping using fixed seeds. For C2, we resample the per-item CAGS with replacement B=1000B = 1000 times, reporting the 2.5 and 97.5 percentiles of the resampled mean distribution. For C4, we resample items within each model-dataset-condition stratum B=2000B = 2000 times. For the human baseline study, we resample items B=5000B = 5000 times (seed 42), employing paired resampling to evaluate comparative differences. For C3 counting and absolute distance metrics, we perform cluster bootstrapping at the source-plate level rather than the individual row level, as multiple spatial questions reference the same underlying photograph. We apply a strict interpretation rule throughout our analysis: if the confidence intervals of two models overlap on a given metric, they are considered statistically tied, meaning raw ranking differences are not inherently meaningful.

Codebase and Directory Structure

Each task directory adheres to a standardized four-stage structure. The first stage synthesizes question-answer pairs from raw annotations. The second stage executes inference across the eighteen models, initiated by a preprocessing step that assembles the manifest and triggers the pre-run audit. The final stage computes metrics and generates summary statistics.

Metric Regeneration

The scoring stage operates as a pure function parameterized by the stored model outputs. Given the released prediction files, every table and metric in this supplement can be regenerated by executing the four component scorers sequentially. Note that the C1 scorer strictly requires the synonym map to be built beforehand; omitting this map silently degrades soft-match evaluation to hard-match evaluation rather than raising an explicit error. This dependency ordering is explicitly documented in the component README.

Extended Limitations

The main manuscript briefly outlines nine limitations. Here, we expand upon those points and detail additional component-level constraints omitted for brevity.

Benchmark-Level Limitations

Exclusive Evaluation of Open-Weight Models. We evaluate 18 open-weight VLMs and no proprietary systems (e.g., GPT-4V, Gemini 1.5 Pro). Consequently, we cannot verify whether frontier closed models exhibit the same behavioral pathologies. This primarily impacts the scaling claims (our largest evaluated model is 38.4B) and the granularity claims, as a significantly larger proprietary model might resolve fine-grained taxonomic ranks that our evaluated models fail to capture. Claims derived from item floors and the adversarial conditions remain robust, as they reflect properties of the evaluation set rather than the model population.

Single Decoding Protocol. Our evaluation relies on greedy decoding without chain-of-thought prompting, few-shot exemplars, or multiple inference attempts. Models capable of self-correction or advanced reasoning modes may achieve higher accuracy than reported. However, we intentionally measure zero-shot, single-pass performance, as this aligns with the compute constraints and operational reality of most automated biodiversity deployments.

Template-Generated Prompts. Every query is synthesized from a template parameterized by an annotation. This introduces the risk that a model might infer the correct answer from linguistic cues rather than visual reasoning. However, the tasks that best separate model performance (e.g., open-ended naming, marker localization, and adversarial pair verification) cannot be deduced from the prompt text, as the answers are inherently visually contingent. This risk is most pronounced in the easier multiple-choice families, particularly category_mcq, which achieves a mean CAGS of .833.

Potential Pretraining Data Contamination. Because our images are sourced from public biodiversity corpora, models may have encountered them during pretraining. It is infeasible to definitively audit the proprietary training mixtures of all 18 models. Critically, the bias introduced by data contamination is strictly upward (inflating scores). Because our primary findings center on where models systematically fail, these below-floor results survive the threat of contamination.

Small Sample Size in Cross-Component Analysis. Eighteen models represent a small statistical sample for a correlation matrix. Consequently, our intervals serve only to distinguish significantly correlated pairs from uncorrelated pairs, rather than establishing a strict ranking among correlations. The block-diagonal structure of the matrix is the primary finding, rather than individual pairwise magnitudes.

C1

Taxonomy Backbone Dependency. Our ground-truth hierarchy strictly follows the GBIF Backbone, which aligns with the Catalogue of Life. If a model accurately answers Reptilia to a class-level question, it is technically reproducing valid textbook Linnaean taxonomy, yet the backbone has structurally evolved. We report the gap between strict and soft matching specifically to quantify this taxonomic lag, rather than unfairly penalizing the model’s absolute accuracy.

Taxonomic Rank Inconsistencies. The order column is natively unpopulated for every reptile record in our source data. Analytical tooling that rigidly assumes six fully populated ranks will silently discard the entire reptile class. To prevent this, our tree structure accommodates skip edges, attaching a node directly to its nearest populated ancestor.

Uniform Distractor Sampling. The distractor pools for multiple-choice questions are unioned across genus, family, order, and class, and then sampled uniformly. Consequently, the class-level set statistically dominates the distractors. The MCQ items should therefore be interpreted as coarse recognition probes. Analysts requiring graded discrimination should condition their evaluation on the provided confusability tiers.

Gymnosperm Representation. There are only roughly one thousand accepted gymnosperm species worldwide. Securing 10 000 images required significantly relaxing our sampling constraints. Notably, gymnosperms represent the sole dissociation in the cross-group correlation matrix and exhibit the lowest loading on the shared C1 factor. We retain and report this group for completeness, but attribute the dissociation to data coverage constraints.

C2

Clade-Specific Attribute Grounding. The CUB-200-2011 dataset consists entirely of avian species. The generalizability of these attribute grounding results to fundamentally different body plans (e.g., invertebrates, mammals) remains untested.

Vocabulary Constraints. Color, pattern, and shape lexicons strictly adhere to the original CUB annotation schema. For example, the limitation to fifteen discrete color words restricts the granularity of the evaluation. Expanding this vocabulary would fundamentally alter the option distributions and the associated chance baselines.

Kernel Dependency in CAGS. While the sensitivity sweep demonstrates ranking stability (ρ≥.9959\rho\ge.9959), absolute CAGS values remain kernel-dependent. A reported CAGS of .593 should therefore be interpreted as relative positioning within the evaluated field, not as a perfectly calibrated absolute probability.

Laterality Bias in the Anatomy Kernel. The metric inherently over-credits lateral proximity: confusing a left wing for a right wing yields a score of .883 due to their spatial closeness in the anatomy layout. This represents the metric’s single systematic divergence from human annotator intuition, which we quantify via the human study. To mitigate this, we compute the laterality analysis using EM criteria and deliberately exclude marker_localization from the primary aggregate score, where this bias would otherwise dominate.

Human Ceiling Estimation Constraints. The human performance ceiling (.908) is measured against a stratified subset of 306 items, not the entire corpus. This figure serves to validate the mathematical metric rather than establishing a definitive human benchmark for the entire dataset.

Approximated Open-Ended Baselines. The chance baseline for open-ended queries is established by sampling a fixed-seed pool of 40 alternate gold answers within that family. While stable under fixed-seed resampling, it remains a statistical approximation of the true full-distribution chance baseline.

Entanglement with C3. The disattenuated correlation between C2 and C3 is 1.015. At this level of statistical reliability, the two components cannot be differentiated by empirical model behavior alone; they are distinguished solely by their semantic task content.

C3

Dataset and Condition Specificity in Relative Position. The relative position task is executed exclusively on the beetle subset under the distractor condition. Lacking a clean (distractor-free) contrast baseline, we cannot statistically isolate the difficulty of the spatial relation reasoning from the visual interference introduced by the distractors.

Corpus Overfitting Risks. Absolute distance, relative distance, and relative position are evaluated solely on the beetle corpora, as this is the only subset possessing the requisite ground truth. Any idiosyncratic properties of this corpus (such as a fixed focal depth or highly uniform backgrounds) are unavoidably confounded with the spatial tasks themselves.

Indicative Nature of Counting Comparisons. Standard benchmarks evaluate counting capabilities using five-option multiple choice over sparse scenes. We challenge models with open numeric generation over dense, heavily occluded scenes of visually homogeneous animals. The observed three-fold performance gap is highly indicative of failure but is not perfectly controlled against standard sparse-scene performance.

OCR Control Image Sharing. The OCR control task utilizes the same beetle trays employed in the spatial evaluations, as these trays naturally feature specimen labels. A control task sourced from a completely independent corpus would provide a more robust decoupled baseline.

C4

Embedding Circularity. The difficulty conditions are explicitly defined by DINOv2 cosine similarity margins. Several evaluated models share architectural lineage or self-supervised training objectives with DINOv2. Consequently, our difficulty manipulation partially indexes shared embedding geometries rather than universal properties of visual identity representation. While we do not replicate the construction across a second embedding space, the MegaDescriptor result tightly bounds this concern: as a system trained on an entirely disparate objective, it exhibits the exact same failure modes.

Absence of a Definitive Performance Ceiling. The specialist model effectively ties, rather than definitively exceeding, the best VLM on neutral pairs. Thus, we have not established the absolute theoretical upper bound for performance on these image pairs. Our claims remain strictly comparative across the evaluated models and conditions.

Single Prompt Formulation. The near-universal bias toward affirmative responses (“yes”) in the hard condition may stem from architectural visual-matching tendencies, or it could be an artifact of our specific prompt wording. Testing a single prompt formulation prevents us from cleanly decoupling the two.

Composite Image Formatting. To test identity, we concatenate pairs into a single side-by-side image rather than passing two separate image tokens. This introduces an auxiliary spatial-parsing burden to the identity judgment. However, because this formatting is held constant across all models and all difficulty conditions, it does not confound the relative difficulty effect, even if it marginally suppresses absolute accuracy levels.

Bias Toward Pattern-Based Identity. The five selected species primarily communicate individual identity through strong pattern-based visual signals. Species whose identification relies strictly on morphological shape variance or incidental scarring remain unrepresented, with the beluga whale serving as the closest conceptual proxy in our set.

Single Operating Point Estimation. Because models output a discrete binary token, we cannot generate a full ROC curve; d′d' is estimated from a single hit/false-alarm pair. While the applied log-linear correction is standard psychophysical practice, it can disproportionately influence estimates in extreme edge-case cells.

Multiple Hypothesis Testing. Evaluating eighteen models across five species and three pairwise contrasts yields several hundred statistical comparisons, guaranteeing that some nominally significant per-cell results will arise by chance. Consequently, per-cell results should be interpreted as exploratory, while the pooled, aggregate tests serve as our confirmatory findings.

Item Non-Independence. Certain source images are utilized as anchors in multiple pairwise comparisons. Bootstrap confidence intervals computed under the assumption of strict independence will therefore be marginally optimistic.

Assumptions Underlying d′d'. Our analysis relies on the assumption of equal-variance Gaussian evidence distributions. We utilize d′d' primarily as a bias-corrected measure of effect size, rather than asserting it acts as a perfect proxy for internal model representations.

Future Directions

We highlight three critical augmentations that would significantly mature this benchmark protocol:

First, evaluating frontier proprietary models (e.g., GPT-4o, Gemini 1.5 Pro) on a stratified subset would conclusively demonstrate whether taxonomic granularity collapse is a universal limitation of current multimodal architectures, or merely a constraint of open-weight parameter scales.

Second, establishing a rigorous human ceiling on a stratified sample of C1 and C3 tasks would allow analysts to benchmark model deficits against genuine domain-expert performance, rather than solely against algorithmic specialists.

Finally, replicating the C4 adversarial pair construction within a disparate, non-DINO embedding space (e.g., CLIP, SigLIP) would definitively close the circularity concern, confirming that the observed failure modes stem from fundamental appearance-matching heuristics rather than shared geometric artifacts.

Additional Tables

The sections above report the statistic that carries each argument. This section reports the rest. Every number here comes from the same scored files. Colors mark the best, second best, second worst and worst over the 18 generalists, and the two shaded specialist rows are excluded from that ranking.

Modelphyclsordfamgenspcscicommcq
BioCLIP-2 kingdom.985.871.816.772.692.547.547.502.966
BioCLIP-2 tol.925.874.821.775.692.545.545.500.966
Qwen3-VL-30B-A3B.882.829.758.615.420.072.233.107.772
Qwen3-VL-32B.861.713.718.615.448.067.258.078.809
Qwen3-VL-8B.895.776.721.592.409.047.213.065.775
InternVL3-38B.932.779.640.488.359.054.199.044.752
Qwen2.5-VL-7B.763.598.674.548.421.047.283.062.753
Gemma-3-12B.928.783.629.431.229.048.101.047.687
InternVL3.5-38B.855.729.625.378.273.038.135.025.707
Gemma-4-12B.880.761.579.365.185.021.070.043.657
InternVL3-8B.903.565.478.330.264.032.131.020.643
Qwen2.5-VL-3B.438.553.477.467.362.040.220.064.689
Llama-3.2-Vision.565.578.519.439.239.031.113.054.655
InternVL3.5-8B.918.622.423.272.193.023.083.015.595
Molmo2-8B.953.644.472.287.142.021.041.016.574
Pixtral-12B.736.579.547.350.191.029.085.027.592
InternVL3.5-30B-A3B.554.558.545.314.242.034.126.024.653
Phi-3.5-Vision.768.424.267.191.069.013.026.009.443
LLaVA-NeXT-Mistral.368.484.409.163.032.005.017.004.465
LLaVA-NeXT-Vicuna.671.435.290.101.032.003.015.003.382

Table 73. EMsoft\mathrm{EM}_{\mathrm{soft}} by rank for every configuration, on the same items as Table 72. The two specialist rows barely move, because a specialist emits current names.

Modelphyclsordfamgenspcscicommcq
BioCLIP-2 kingdom.985.871.816.772.691.547.546.502.966
BioCLIP-2 tol.925.874.821.775.691.545.544.500.966
Qwen3-VL-30B-A3B.864.731.757.612.417.072.221.107.772
Qwen3-VL-32B.831.615.712.612.444.066.244.078.809
Qwen3-VL-8B.887.680.720.589.406.046.202.065.775
InternVL3-38B.911.683.638.484.355.054.189.044.752
Qwen2.5-VL-7B.763.504.673.541.420.047.269.062.753
Gemma-3-12B.790.688.626.429.225.048.093.047.687
InternVL3.5-38B.746.636.624.363.271.038.128.025.707
Gemma-4-12B.880.686.578.363.183.021.065.043.657
InternVL3-8B.827.497.478.322.262.032.124.020.643
Qwen2.5-VL-3B.416.457.477.454.358.040.206.064.689
Llama-3.2-Vision.545.482.518.434.234.031.104.054.655
InternVL3.5-8B.819.531.421.266.191.023.077.015.595
Molmo2-8B.823.563.469.286.137.021.038.016.574
Pixtral-12B.688.501.530.349.187.029.079.027.592
InternVL3.5-30B-A3B.486.469.545.310.240.034.118.024.653
Phi-3.5-Vision.748.359.267.191.066.013.023.009.443
LLaVA-NeXT-Mistral.364.392.408.162.029.005.015.004.465
LLaVA-NeXT-Vicuna.662.343.290.100.028.003.015.003.382

Table 72. EM by rank for every configuration. Table 19 gives the column means. Every model falls monotonically from phylum to the species epithet, and the common-name MCQ column sits far above the open common-name column for all 20.

Modelphyclsordfamgenspcscicommcq
BioCLIP-2 kingdom.000.000.000.000.001.000.001.000.000
BioCLIP-2 tol.000.000.000.000.001.000.001.000.000
Qwen3-VL-30B-A3B.018.098.001.003.003.000.012.000.000
Qwen3-VL-32B.030.098.006.003.004.001.014.000.000
Qwen3-VL-8B.008.096.001.003.003.001.011.000.000
InternVL3-38B.021.096.002.004.004.000.010.000.000
Qwen2.5-VL-7B.000.094.001.007.001.000.014.000.000
Gemma-3-12B.138.095.003.002.004.000.008.000.000
InternVL3.5-38B.109.093.001.015.002.000.007.000.000
Gemma-4-12B.000.075.001.002.002.000.005.000.000
InternVL3-8B.076.068.000.008.002.000.007.000.000
Qwen2.5-VL-3B.022.096.000.013.004.000.014.000.000
Llama-3.2-Vision.020.096.001.005.005.000.009.000.000
InternVL3.5-8B.099.091.002.006.002.000.006.000.000
Molmo2-8B.130.081.003.001.005.000.003.000.000
Pixtral-12B.048.078.017.001.004.000.006.000.000
InternVL3.5-30B-A3B.068.089.000.004.002.000.008.000.000
Phi-3.5-Vision.020.065.000.000.003.000.003.000.000
LLaVA-NeXT-Mistral.004.092.001.001.003.000.002.000.000
LLaVA-NeXT-Vicuna.009.092.000.001.004.000.000.000.000

Table 74. EMsoft_{\mathrm{soft}} − EM by rank, the difference between the two tables above. The gain concentrates at class and phylum for nearly every model and is zero from the species epithet rightward. Table 23 pools this by rank and by group.

ModelAmphAngioArachBirdsBryoFishGymnoInsectMammRept
BioCLIP-2 kingdom.766.875.781.887.659.719.215.921.819.795
BioCLIP-2 tol.765.864.777.886.653.718.182.921.817.793
Qwen3-VL-30B-A3B.495.561.497.691.322.345.482.605.614.406
Qwen3-VL-32B.485.536.525.653.328.334.388.645.567.407
Qwen3-VL-8B.460.595.478.645.285.327.469.601.574.385
InternVL3-38B.457.480.444.640.265.297.483.573.552.346
Qwen2.5-VL-7B.425.454.459.636.264.278.488.573.506.362
Gemma-3-12B.420.367.382.539.256.229.470.487.544.310
InternVL3.5-38B.400.332.408.555.239.205.400.528.521.306
Gemma-4-12B.361.453.411.474.211.194.516.468.478.263
InternVL3-8B.385.310.403.497.183.206.320.480.448.309
Qwen2.5-VL-3B.334.361.421.512.255.212.298.522.357.212
Llama-3.2-Vision.324.280.430.544.148.154.310.484.466.225
InternVL3.5-8B.368.293.359.474.156.195.266.445.447.243
Molmo2-8B.387.297.396.463.156.192.190.437.468.241
Pixtral-12B.312.289.395.480.196.219.280.460.442.207
InternVL3.5-30B-A3B.328.290.400.420.168.221.274.471.357.234
Phi-3.5-Vision.199.206.371.319.061.148.198.338.345.156
LLaVA-NeXT-Mistral.203.104.354.342.047.151.116.290.312.096
LLaVA-NeXT-Vicuna.215.066.356.329.073.144.071.304.295.160

Table 75. EM by organismal group for every configuration. Birds and Insects are the easiest groups for almost every model, Bryophytes the hardest. Gymnosperms is the one column where the specialist loses to most generalists, which is the same dissociation Table 24 reports.

ModelEasyMediumHardDegen.
BioCLIP-2 kingdom.800.698.616.985
BioCLIP-2 tol.798.699.619.925
Qwen3-VL-30B-A3B.606.429.295.864
Qwen3-VL-32B.613.428.245.831
Qwen3-VL-8B.592.412.246.887
InternVL3-38B.562.375.207.911
Qwen2.5-VL-7B.542.408.227.763
Gemma-3-12B.499.330.185.790
InternVL3.5-38B.501.331.160.746
Gemma-4-12B.468.307.148.880
InternVL3-8B.444.300.098.827
Qwen2.5-VL-3B.489.332.161.416
Llama-3.2-Vision.463.297.131.545
InternVL3.5-8B.423.244.077.819
Molmo2-8B.432.240.061.823
Pixtral-12B.432.270.107.688
InternVL3.5-30B-A3B.456.280.106.486
Phi-3.5-Vision.284.152.041.748
LLaVA-NeXT-Mistral.300.174.039.364
LLaVA-NeXT-Vicuna.245.128.029.662

Table 76. EM by difficulty tier for every configuration. All 20 satisfy Easy > Medium > Hard without exception. The Degenerate column holds items in cells too thin to tier, described in Sec. E.

RankEasyMediumHard
class.796.691.233
order.786.605.299
family.580.387.270
genus.444.265.165
species epithet.101.083.074
scientific name.236.148.116
common name.113.075.065
common name MCQ.705.618.573

Table 77. Mean EM by rank and difficulty tier, pooled over all 20 configurations. The tier gap is widest at class and order and narrowest at the species epithet, where every tier is already near the floor. Phylum carries no tier, because a single group holds too few phyla to cut tertiles.

Modelopen-ended4-way MCQgap
Qwen3-VL-32B.078.809+.731
Qwen3-VL-8B.065.775+.710
InternVL3-38B.044.752+.708
Qwen2.5-VL-7B.062.753+.691
InternVL3.5-38B.025.707+.682
Qwen3-VL-30B-A3B.107.772+.665
Gemma-3-12B.047.687+.640
InternVL3.5-30B-A3B.024.653+.629
Qwen2.5-VL-3B.064.689+.625
InternVL3-8B.020.643+.623
Gemma-4-12B.043.657+.614
Llama-3.2-Vision.054.655+.601
InternVL3.5-8B.015.595+.580
Pixtral-12B.027.592+.565
Molmo2-8B.016.574+.558
BioCLIP-2 tol.500.966+.466
BioCLIP-2 kingdom.502.966+.464
LLaVA-NeXT-Mistral.004.465+.461
Phi-3.5-Vision.009.443+.434
LLaVA-NeXT-Vicuna.003.382+.379

Table 78. Open-ended against four-way multiple choice on the common name, all 20 configurations, sorted by gap. Every configuration recognizes far better than it recalls, including both specialists. Table 20 gives the five widest and the two means.

Modelopen-ended4-way MCQgap
Qwen3-VL-32B.078.809+.732
Qwen3-VL-8B.065.775+.710
InternVL3-38B.044.752+.708
Qwen2.5-VL-7B.062.753+.692
Qwen3-VL-30B-A3B.107.772+.665
18-model mean.039.645+.606
20-config mean.085.677+.592

Table 20. We show models the identical image and ask for the identical organism. They recognize the correct name among four options far more often than they produce it unprompted. For the 18 VLMs, the ratio of mean accuracies is 16.4×.

ModelunresolvedseverityEM
LLaVA-NeXT-Mistral.2332.739.203
LLaVA-NeXT-Vicuna.2203.135.202
Llama-3.2-Vision.2172.493.338
Phi-3.5-Vision.1792.992.235
Qwen2.5-VL-3B.1772.221.350
InternVL3-8B.1602.692.355
InternVL3.5-30B-A3B.1352.667.317
Molmo2-8B.1222.746.324
Pixtral-12B.1212.775.329
Qwen2.5-VL-7B.1102.444.445
InternVL3.5-38B.1082.606.390
Gemma-4-12B.1072.763.384
Qwen3-VL-32B.1022.459.488
Gemma-3-12B.1022.574.401
InternVL3.5-8B.1002.740.325
Qwen3-VL-8B.0882.480.483
InternVL3-38B.0832.614.455
Qwen3-VL-30B-A3B.0812.472.503
BioCLIP-2 tol.0262.638.737
BioCLIP-2 kingdom.0252.523.743

Table 79. Unresolved share, severity and EM for every configuration, sorted by unresolved. An unresolved answer names no node in the tree, so it carries no severity. The share correlates with EM at ρ=−0.854\rho= -0.854.

ModelEMsevEM rksev rkshift
BioCLIP-2 kingdom.7432.52317−6
BioCLIP-2 tol.7372.638211−9
Qwen3-VL-30B-A3B.5032.47234−1
Qwen3-VL-32B.4882.45943+1
Qwen3-VL-8B.4832.48055+0
InternVL3-38B.4552.614610−4
Qwen2.5-VL-7B.4452.44472+5
Gemma-3-12B.4012.57488+0
InternVL3.5-38B.3902.60699+0
Gemma-4-12B.3842.7631017−7
InternVL3-8B.3552.6921113−2
Qwen2.5-VL-3B.3502.221121+11
Llama-3.2-Vision.3382.493136+7
Pixtral-12B.3292.7751418−4
InternVL3.5-8B.3252.7401515+0
Molmo2-8B.3242.7461616+0
InternVL3.5-30B-A3B.3172.6671712+5
Phi-3.5-Vision.2352.9921819−1
LLaVA-NeXT-Mistral.2032.7391914+5
LLaVA-NeXT-Vicuna.2023.1352020+0

Table 80. EM rank against severity rank for every configuration. A positive shift means the model ranks better on severity than on accuracy. Qwen2.5-VL-3B moves 11 places and is the clearest case of a model that is inaccurate and still misses closely.

ModelFamilyParamsEMsev
Qwen2.5-VL-3BQwen2.5-VL3.0.3502.221
Phi-3.5-VisionPhi-3.54.2.2352.992
Qwen2.5-VL-7BQwen2.5-VL7.0.4452.444
LLaVA-NeXT-MistralLLaVA-1.67.0.2032.739
LLaVA-NeXT-VicunaLLaVA-1.67.0.2023.135
InternVL3.5-8BInternVL3.58.0.3252.740
InternVL3-8BInternVL38.0.3552.692
Qwen3-VL-8BQwen3-VL8.0.4832.480
Molmo2-8BMolmo28.0.3242.746
Llama-3.2-VisionLlama-3.211.0.3382.493
Gemma-4-12BGemma-412.0.3842.763
Gemma-3-12BGemma-312.0.4012.574
Pixtral-12BPixtral12.0.3292.775
Qwen3-VL-30B-A3BQwen3-VL30.0.5032.472
InternVL3.5-30B-A3BInternVL3.530.0.3172.667
Qwen3-VL-32BQwen3-VL32.0.4882.459
InternVL3-38BInternVL338.0.4552.614
InternVL3.5-38BInternVL3.538.0.3902.606

Table 81. Parameters against pooled EM and severity for the 18 generalists, sorted by size. Within Qwen3-VL, 8B scores .483 and 32B scores .488. Within InternVL3, 8B scores .355 and 38B scores .455. Scale helps one family and not the other, and it never closes the gap to the specialist.

Metricρ\rho with EMreading
Wu–Palmer+0.959near-redundant with EM
Lin+0.965near-redundant with EM
severity−0.662orders the field differently
unresolved−0.854tracks EM, reported beside it

Table 82. Spearman correlation of each C1 metric with EM over the 20 configurations. Wu–Palmer and Lin reorder almost nothing, so we keep them as corroboration. Severity is the one metric that carries information EM does not.

Modelattr_bareattr_groupcat_mcqmarkercommongenuskind_ofscivis_setvis_side
Qwen3-VL-32B1.0031.0151.000.968.503.995.374.769.8571.069
Qwen3-VL-30B-A3B1.001.9931.000.919.522.977.398.775.8331.034
InternVL3-38B.9981.0081.000.983.592.990.442.777.8141.031
Gemma-4-12B1.0311.0321.0001.078.7901.005.470.939.7631.087
Qwen3-VL-8B1.0101.0061.000.940.555.985.411.765.8611.049
InternVL3.5-38B1.0191.0121.000.951.695.987.593.832.8811.071
Molmo2-8B1.0071.0151.000.949.7611.003.473.921.967.878
Qwen2.5-VL-7B1.0061.0171.000.916.495.990.393.803.8091.022
InternVL3.5-30B-A3B.9891.0171.000.986.770.992.475.850.8781.035
InternVL3-8B.9941.0431.000.991.702.987.441.811.8581.009
InternVL3.5-8B.9761.0021.000.905.7791.004.495.866.7561.055
Gemma-3-12B.993.9961.000.899.6491.001.392.874.999.936
Qwen2.5-VL-3B.968.9851.000.981.5571.001.453.7801.0221.000
Pixtral-12B.9941.0241.000.889.652.992.522.903.8201.150
LLaVA-NeXT-Vicuna.999.9621.000.993.8681.003.796.969.759.721
Phi-3.5-Vision1.0041.0131.000.975.8451.007.918.9791.0881.024
Llama-3.2-Vision.9711.0161.000.926.5751.004.602.8691.0481.003
LLaVA-NeXT-Mistral.9591.0261.000.958.8581.007.701.9571.0521.024
mean.9961.0101.000.956.676.996.519.858.8931.011

Table 83. Mistake severity by question family, the mean 1−CAGS1-\mathrm{CAGS} over wrong items only. Lower is better. A value above 1 means the average miss lands farther from the gold than a random distractor. Table 30 gives the pooled column. Colors mark the best, second best, second worst and worst in each family.

Modelattr_bareattr_groupcat_mcqmarkercommongenuskind_ofscivis_setvis_side
Qwen3-VL-32B.725.721.959.787.624.699.766.655.480.613
Qwen3-VL-30B-A3B.714.697.951.755.613.426.737.622.479.554
InternVL3-38B.724.704.948.721.501.638.670.641.423.572
Gemma-4-12B.710.728.772.756.294.131.621.168.432.575
Qwen3-VL-8B.706.681.958.732.549.489.695.584.475.546
InternVL3.5-38B.724.704.920.754.396.422.524.492.366.584
Molmo2-8B.721.660.532.823.291.154.583.189.419.807
Qwen2.5-VL-7B.670.688.962.720.631.639.739.642.383.507
InternVL3.5-30B-A3B.694.688.869.700.309.510.611.473.388.481
InternVL3-8B.673.672.884.689.381.527.647.533.389.480
InternVL3.5-8B.685.656.783.697.284.354.562.388.399.524
Gemma-3-12B.646.639.914.716.444.304.692.295.472.468
Qwen2.5-VL-3B.599.610.926.632.572.570.669.625.461.504
Pixtral-12B.597.632.847.690.402.266.618.288.383.386
LLaVA-NeXT-Vicuna.564.625.550.608.150.038.240.122.409.525
Phi-3.5-Vision.634.633.716.638.186.076.107.115.203.402
Llama-3.2-Vision.583.547.859.690.628.195.707.305.335.443
LLaVA-NeXT-Mistral.585.584.636.671.163.033.344.128.289.493
mean.664.659.833.710.412.360.585.404.399.526

Table 84. Raw graded similarity GSrawGS_{\mathrm{raw}} by question family, before the chance adjustment. Reading this against Table 33 shows what the adjustment removes. The two are close on the open families, where distractors are sampled and bb is small, and far apart on visibility-partset and attr_group-value, where the option sets sit close to the gold and a raw score flatters every model.

ModelWhale sharkHyenaLeopardGrevy’s zebraBeluga
Qwen3-VL-32B.496.814.517.9311.027
InternVL3-8B.6111.097.981.798.882
Qwen2.5-VL-3B.2871.0481.243.203.470
InternVL3.5-30B-A3B.412.968.443.506.700
Phi-3.5-Vision.364.880.896.344.709
Qwen3-VL-30B-A3B.332.861.5931.088.061
Qwen3-VL-8B.352.780.417.516.821
Molmo2-8B.232.963.644.472.228
LLaVA-NeXT-Mistral.1791.052.7140.000.633
InternVL3-38B.234.729.564.519.337
Qwen2.5-VL-7B.381.709.323.338.332
Gemma-4-12B.399.564.387.670−.042
InternVL3.5-38B.126.287.695.409.335
InternVL3.5-8B.398.401−.075.204.294
Gemma-3-12B.501.143.413−.379.172
Pixtral-12B.077.074.141.127.441
Llama-3.2-Vision−.406.728.414.286−.499
LLaVA-NeXT-Vicuna0.0000.0000.0000.0000.000

Table 86. d′d' by model and species under random pairing, the neutral condition. Table 52 gives the column means. No model reaches d′=1.5d' = 1.5 on any species, and beluga is the hardest column for almost every model.

Modelaccbal. accMCCκ\kappa
Qwen3-VL-32B.636.636.284.272
InternVL3-8B.601.601.256.202
Qwen2.5-VL-3B.559.559.176.119
InternVL3.5-30B-A3B.612.612.231.223
Phi-3.5-Vision.576.576.198.151
Qwen3-VL-30B-A3B.602.602.216.203
Qwen3-VL-8B.572.572.181.145
Molmo2-8B.578.578.179.157
LLaVA-NeXT-Mistral.508.508.080.016
InternVL3-38B.581.581.177.162
Qwen2.5-VL-7B.541.541.121.082
Gemma-4-12B.567.567.147.134
InternVL3.5-38B.542.542.107.084
InternVL3.5-8B.544.544.094.088
Gemma-3-12B.505.505.028.009
Pixtral-12B.519.519.054.038
Llama-3.2-Vision.500.500.019.000
LLaVA-NeXT-Vicuna.500.500nan0.000

Table 87. Accuracy, balanced accuracy, Matthews correlation and Cohen’s κ\kappa under random pairing, averaged over the five species. We report d′d' as the headline because these four all conflate discrimination with response bias, and Table 85 shows the bias is large. The four agree with each other and with d′d' on the broad ordering.

Modeleasyrandomhard
Qwen3-VL-32B−.270−.076−.382
InternVL3-8B−.652−.765−1.024
Qwen2.5-VL-3B−.843−1.264−1.440
InternVL3.5-30B-A3B−.003−.066−.263
Phi-3.5-Vision−.740−.885−1.133
Qwen3-VL-30B-A3B−.010.118−.052
Qwen3-VL-8B−.611−.776−.986
Molmo2-8B−.378−.660−.839
LLaVA-NeXT-Mistral−2.221−2.241−2.301
InternVL3-38B−.303−.494−.767
Qwen2.5-VL-7B−1.020−1.242−1.352
Gemma-4-12B−.257−.292−.434
InternVL3.5-38B−.397−.640−.759
InternVL3.5-8B−.170−.355−.401
Gemma-3-12B−1.257−1.818−1.767
Pixtral-12B−.570−.665−.696
Llama-3.2-Vision−.999−1.340−1.429
LLaVA-NeXT-Vicuna−3.540−3.540−3.540

Table 85. Criterion cc for every model in all three conditions, pooled over the five species by summing the confusion counts. Negative means a standing preference for yes. Colors mark the smallest and largest ∣c∣|c|, so best is the least biased. Every model leans toward yes, and 17 of 18 lean further as the pairs harden.

Modeleasyhardeasy − hard
InternVL3-8B80.141.338.8
InternVL3-38B77.739.338.4
InternVL3.5-30B-A3B78.339.938.4
Phi-3.5-Vision77.140.137.0
Qwen3-VL-32B75.938.936.9
Molmo2-8B80.244.236.0
Qwen2.5-VL-3B79.844.535.3
Qwen3-VL-8B75.342.832.5
Gemma-4-12B73.642.531.1
InternVL3.5-38B72.343.129.1
Qwen3-VL-30B-A3B73.945.128.8
InternVL3.5-8B67.244.722.5
Qwen2.5-VL-7B65.846.819.0
Pixtral-12B60.946.214.7
Llama-3.2-Vision60.548.512.0
Gemma-3-12B58.749.69.1
LLaVA-NeXT-Mistral54.449.25.2
LLaVA-NeXT-Vicuna50.050.00.0

Table 88. The difficulty manipulation for every model, in percentage points of accuracy averaged over the five species. Seventeen models confirm the ordering, one is flat, and none reverses. The flat model emits a constant answer and cannot respond to any manipulation, which is what makes it a useful control.

References

  1. [1]Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.arxiv.org/abs/2404.14219
  2. [2]Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tal-lyQA: Answering complex counting questions. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8076–8084, 2019.
  3. [3]Allen Institute for AI. Molmo 2. https://huggingface.co/allenai/Molmo2-8B, 2025. Released 16 December 2025; accessed 27 July 2026.
  4. [4]BeeLivingSensor project. Boxes on bees and pollen. LILA BC, https://lila.science/datasets/boxes-on-bees-and-pollen, 2021. Community Data License Agreement.
  5. [5]Cigdem Beyan, Anil Osman Tur, and Ehsan Karimi. From species-specific models to universal re-ID: A survey of animal re-identification. Information Fusion, page 104323, 2026.DOI
  6. [6]Vojtěch Čermák, Lukáš Picek, Lukáš Adam, and Kostas Papafitsoros. WildlifeDatasets: An open-source toolkit for animal re-identification. In WACV, 2024. Introduces MegaDescriptor.arxiv.org/abs/2311.09118
  7. [7]Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2401.12168
  8. [8]Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960.DOI
  9. [9]James P. Crall, Charles V. Stewart, Tanya Y. Berger-Wolf, Daniel I. Rubenstein, and Siva R. Sundaresan. HotSpotter — patterned species instance recognition. In WACV, 2013.DOI
  10. [10]Alyson East, Elizabeth G. Campolongo, Luke Meyers, S M Rayeed, Samuel Stevens, Iuliia Zarubieva, Isadora E. Fluck, Jennifer C. Girón, Maximiliane Jousse, Scott Lowe, Kayla I. Perry, Isabelle Betancourt, Noah Charney, Evan Donoso, Nathan Fox, Kim J. Landsbergen, Ekaterina Nepovinnykh, Michelle Ramirez, Parkash Singh, Khum Thapa-Magar, Matthew Thompson, Evan Waite, Tanya Berger-Wolf, Hilmar Lapp, Paula Mabee, Charles Stewart, Graham Taylor, and Sydne Record. Optimizing image capture for computer vision-powered taxonomic identification and trait recognition of biodiversity specimens. Methods in Ecology and Evolution, 16(10):2260–2275, 2025.DOI
  11. [11]trap images. arXiv preprint arXiv:2311.01064, 2023. Introduces WildMatch.arxiv.org/abs/2311.01064
  12. [12]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In ECCV, 2024.arxiv.org/abs/2404.12390
  13. [13]GBIF Secretariat. GBIF backbone taxonomy. Darwin Core Archive, 2024. Accessed 27 July 2026.
  14. [14]Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
  15. [15]Gemma Team, Google DeepMind. Gemma 4: Byte for byte, the most capable open models. https://blog.google/innovation-and-ai/technology/developers-tools/google-gemma-4/, 2026. Released 2 April 2026; accessed 27 July 2026.
  16. [16]Simon Ging, María A. Bravo, and Thomas Brox. Open-ended VQA benchmarking of vision-language models by exploiting classification datasets and their semantic hierarchy. In ICLR, 2024.arxiv.org/abs/2402.07270
  17. [17]David M. Green and John A. Swets. Signal Detection Theory and Psychophysics. Wiley, 1966.
  18. [18]Jianyong Gu, Samuel Stevens, Elizabeth G. Campolongo, Matthew J. Thompson, Net Zhang, Jiaman Wu, Andrei Kopanev, Zheda Mai, Alexander E. White, James Balhoff, Wasila Dahdul, Daniel Rubenstein, Hilmar Lapp, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su. BioCLIP 2: Emergent properties from scaling hierarchical contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 102778–102811, 2025. Introduces the TreeOfLife-200M corpus.
  19. [19]Michael J. Hautus. Corrections for extreme proportions and their biasing effects on estimated values of d′. Behavior Research Methods, Instruments, & Computers, 27(1):46–51, 1995.DOI
  20. [20]Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. TransReID: Transformer-based object re-identification. In ICCV, 2021.arxiv.org/abs/2102.04378
  21. [21]Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, 2007.
  22. [22]Imageomics Institute. 2018-NEON-beetles. Hugging Face Datasets, https://huggingface.co/datasets/imageomics/2018-NEON-beetles, 2023. Specimens from the National Ecological Observatory Network ground-beetle pitfall-trap product; elytra measurements annotated through Zooniverse.
  23. [23]iNaturalist. iNaturalist. https://www.inaturalist.org, 2026. Taxon and vernacular-name records retrieved via the iNaturalist API, https://api.inaturalist.org/v1/docs/; accessed 27 July 2026.
  24. [24]Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking multimodal LLMs with generative comprehension. In CVPR, 2024.arxiv.org/abs/2307.16125
  25. [25]Dekang Lin. An information-theoretic definition of similarity. In ICML, 1998.
  26. [26]Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11, 2023.
  27. [27]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.arxiv.org/abs/2310.03744
  28. [28]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? In ECCV, 2024.arxiv.org/abs/2307.06281
  29. [29]Llama Team, AI @ Meta. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  30. [30]Neil A. Macmillan and C. Douglas Creelman. Detection Theory: A User’s Guide. Lawrence Erlbaum Associates, 2nd edition, 2004.
  31. [31]Zheda Mai, Arpita Chowdhury, Zihe Wang, Sooyoung Jeon, Lemeng Wang, Jiacheng Hou, Jihyung Kil, and Wei-Lun Chao. AVA-Bench: Atomic visual ability benchmark for vision foundation models. In CVPR, 2026.arxiv.org/abs/2506.09082
  32. [32]M. Maruf, Arka Daw, Kazi Sajeed Mehrab, Harish Babu Manogaran, Abhilash Neog, Medha Sawhney, Mridul Khurana, James P. Balhoff, Yasin Bakis, Bahadir Altintas, Matthew J. Thompson, Elizabeth G. Campolongo, Josef C. Uyeda, Hilmar Lapp, Henry L. Bart, Paula M. Mabee, Yu Su, Wei-Lun Chao, Charles Stewart, Tanya Berger-Wolf, Wasila Dahdul, and Anuj Karpatne. VLMBio: A benchmark dataset to evaluate pretrained vision-language models for trait discovery from biological images. In NeurIPS Datasets and Benchmarks, 2024.
  33. [33]Zhongqi Miao, Yuanhan Zhang, Zalan Fabian, Andres Hernandez Celis, Sara Beery, Chunyuan Li, Ziwei Liu, Amrita Gupta, Md Nasir, Wanhua Li, Jason Holmberg, Meredith Palmer, Kaitlyn Gaynor, Pablo Arbelaez, Pengce Wang, Rahul Dodhia, and Juan Lavista Ferres. New frontiers in artificial intelligence for biodiversity research and conservation with multimodal language models. Methods in Ecology and Evolution, 2025.DOI
  34. [34]Mistral AI. Pixtral 12B. arXiv preprint arXiv:2410.07073, 2024.
  35. [35]Olga Moskvyak, Frederic Maire, Feras Dayoub, Asia O. Armstrong, and Mahsa Baktashmotlagh. Robust re-identification of manta rays from natural markings by learning pose invariant embeddings. In International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 1–8, 2021.arxiv.org/abs/1902.10847
  36. [36]Chao Mou, Tengfei Liu, Chengcheng Zhu, and Xiaohui Cui. WAID: A large-scale dataset for wildlife detection with drones. Applied Sciences, 13(18):10397, 2023.DOI
  37. [37]Ekaterina Nepovinnykh, Ilia Chelak, Tuomas Eerola, Veikka Immonen, Heikki Kälviäinen, Maksim Kholiavchenko, and Charles V. Stewart. Species-agnostic patterned animal re-identification by aggregating deep local features. International Journal of Computer Vision, 2024.DOI
  38. [38]Ekaterina Nepovinnykh, Ilan Beyen, Aidan Westphal, Ankit K. Upadhyay, S M Rayeed, Julian Bain, Jason Parham, Jonathan Crall, David Kimiti, Belinda Mackey, Daniel Rubenstein, Tanya Berger-Wolf, Timm Haucke, Sara Beery, Tuomas Eerola, Heikki Kälviäinen, and Charles V. Stewart. Species-agnostic patterned animal re-identification by aggregating deep local features. International Journal of Computer Vision, 2024.
  39. [39]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2304.07193
  40. [40]Lasha Otarashvili, T. Subramanian, Jason Holmberg, J. J. Levenson, and Charles V. Stewart. Multispecies animal re-ID using a large community-curated dataset. arXiv preprint arXiv:2412.05602, 2024. MiewID; 49 species, 59 datasets, 225,374 annotations.arxiv.org/abs/2412.05602
  41. [41]Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching CLIP to count to ten. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 3170–3180, 2023.arxiv.org/abs/2302.12066
  42. [42]Qwen Team. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025.
  43. [43]Qwen Team. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.
  44. [44]S M Rayeed, Alyson East, Samuel Stevens, Sydne Record, and Charles V. Stewart. Beetleverse: A study on taxonomic classification of ground beetles. arXiv preprint arXiv:2504.13393, 2025.arxiv.org/abs/2504.13393
  45. [45]S M Rayeed, Alyson East, Samuel Stevens, Sydne Record, and Charles V. Stewart. Fine-grained beetle taxonomy with vision models: A benchmark on long-tailed and domain-adaptive classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 5093–5099, 2025.
  46. [46]S M Rayeed, Mridul Khurana, Alyson East, Isadora E. Fluck, Elizabeth G. Campolongo, Samuel Stevens, Iuliia Zarubieva, Scott C. Lowe, Michael W. Denslow, Evan D. Donoso, Jiaman Wu, Michelle Ramirez, Benjamin Baiser, Charles V. Stewart, Paula Mabee, Tanya Berger-Wolf, Anuj Karpatne, Hilmar Lapp, Robert P. Guralnick, Graham W. Taylor, and Sydne Record. A continental-scale dataset of ground beetles with high-resolution images and validated morphological trait measurements. arXiv preprint arXiv:2601.10687, 2026.arxiv.org/abs/2601.10687
  47. [47]Syed Nazmus Sakib, Nafiul Haque, Mohammad Zabed Hossain, and Shifat E. Arman. PlantExpertVQA: A visual question answering dataset for benchmarking vision-language models in plant science. arXiv preprint arXiv:2508.17117, 2025.arxiv.org/abs/2508.17117
  48. [48]Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design. In ICLR, 2024.
  49. [49]Samuel Stevens, Jiaman Wu, Matthew J. Thompson, Elizabeth G. Campolongo, Chan Hee Song, David E. Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su. BioCLIP: A vision foundation model for the tree of life. In CVPR, 2024.arxiv.org/abs/2311.18803
  50. [50]Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middlegu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In NeurIPS, 2024. Introduces CV-Bench.arxiv.org/abs/2406.16860
  51. [51]Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In CVPR, 2024.arxiv.org/abs/2401.06209
  52. [52]Shinya Uryu. Evaluating large language models for IUCN red list species information. arXiv preprint arXiv:2510.02830, 2025.arxiv.org/abs/2510.02830
  53. [53]Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate E. Jones, Oisin Mac Aodha, Sara Beery, and Grant Van Horn. INQUIRE: A natural world text-to-image retrieval benchmark. In NeurIPS Datasets and Benchmarks, 2024.arxiv.org/abs/2411.02537
  54. [54]M. Vijayalakshmi and A. Sasithradevi. A comprehensive annotated image dataset for real-time fish detection in pond settings. Data in Brief, 57:111007, 2024. Introduces the DePondFi (Detection of Pond Fish) dataset.
  55. [55]Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
  56. [56]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025.
  57. [57]Yibin Wen, Qingmei Li, Ziye, Jiarui Zhang, Xiaoya Fan, Zurong Mai, Jing Wu, Shuohong Lou, Yuhang Chen, Henglian Huang, Yang Zhang, Defeng Gu, Lingyuan Zhao, Yutong Lu, Haohuan Fu, Jianxi Huang, and Juepeng Zheng. AgroCoT: A chain-of-thought benchmark for evaluating reasoning in vision-language models for agriculture. arXiv preprint arXiv:2511.23253, 2025.
  58. [58]Wikimedia Foundation. Wikidata: A free collaborative knowledge base. https://www.wikidata.org, 2026. Vernacular-name records retrieved via the Wikidata query service; accessed 27 July 2026.
  59. [59]Wild Me. Whale shark ID. LILA BC, https://lila.science/datasets/whale-shark-id, 2020. Whale shark (Rhincodon typus), Ningaloo Marine Park.
  60. [60]Wild Me. Beluga ID 2022. LILA BC, https://lila.science/datasets/beluga-id-2022/, 2022. Community Data License Agreement (permissive variant).
  61. [61]Wild Me. Hyena ID 2022. LILA BC, https://lila.science/datasets/hyena-id-2022/, 2022. Spotted hyena (Crocuta crocuta); collected with the Botswana Predator Conservation Trust.
  62. [62]Wild Me. Leopard ID 2022. LILA BC, https://lila.science/datasets/leopard-id-2022/, 2022. African leopard (Panthera pardus); collected with the Botswana Predator Conservation Trust.
  63. [63]Zhibiao Wu and Martha Palmer. Verb semantics and lexical selection. In ACL, pages 133–138, 1994.
  64. [64]Chih-Hsuan Yang, Benjamin Feuer, Talukder Zaki Jubery, Zi Deng, Andre Nakkab, Md Zahid Hasan, Shivani Chiranjeevi, Kelly Marshall, Nirmal Baishnab, Asheesh K. Singh, Arti Singh, Soumik Sarkar, Nirav Merchant, Chinmay Hegde, and Baskar Ganapathy Subramanian. BioTrove: A large curated image dataset enabling AI for biodiversity. In Advances in Neural Information Processing Systems 37 (NeurIPS) Datasets and Benchmarks Track, 2024.arxiv.org/abs/2406.17720
  65. [65]Takero Yoshida, Yuikazu Ito, Yoshihiro Fujiwara, Shinji Tsuchida, Daisuke Sugiyama, and Daisuke Matsuoka. J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM. arXiv preprint arXiv:2412.15574, 2024.arxiv.org/abs/2412.15574
  66. [66]Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.arxiv.org/abs/2504.10479

Paper details

vlnc:W_mkf76wem
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents