betaFeedback

WorksComputer Science

RankShift: Evaluating Ranking Instability Across Benchmark Dimensions in Document Parsing

Abstract

Document parsing benchmarks are often treated as stable measures of model quality, yet rankings may depend substantially on benchmark design choices. In this work, we present RankShift, a systematic study comparing OCR and vision-language models (VLMs) under controlled variation in dataset composition, matching strategy, and scoring methodology. Using OmniDocBench as a case study, we evaluate the ranking stability of 16 models across 3 dataset configurations, 6 matching strategies, and 3 scoring metrics. We quantify rank stability using pairwise rank correlation, top-k churn, and pairwise reversal rate. Our results show that the interaction between matching strategy and ground truth format is a substantial but understudied driver of instability: scoring against OmniDocBench's JSON annotations versus its Markdown rendering yields a mean Kendall's tau of approximately 0.33 under region-level matching strategies, with individual pairs as low as tau = 0.23, compared with tau = 0.77 when each page is scored as a single concatenated string. Dataset composition produces comparable instability across page-type strata (mean tau = 0.48) and layout-type strata (mean tau = 0.49), while the cross-dataset shift from OmniDocBench to Real5's Scanning reconstruction of the same pages is more stable (tau = 0.80). Metric choice is the most stable dimension, with mean tau of approximately 0.78 across NED, BLEU, and METEOR. These findings indicate that leaderboard position is not solely a property of model capability, motivating leaderboards that report not only headline scores but also the robustness of rankings across reasonable design choices.

Introduction

Benchmarks are essential tools for tracking progress in machine learning, yet they are often treated as static and objective arbiters of model capability [4, 28, 10]. In practice, benchmarks encode a range of design choices, including dataset composition, pre- and post-processing steps, and scoring methodology. The effects of these decisions on model rankings are rarely made explicit and remain largely understudied in document parsing. As benchmarks become increasingly saturated and susceptible to overfitting, these choices take on greater significance: small differences in evaluation design can determine a model’s overall ranking, independent of genuine capability differences.

Document parsing, the task of converting unstructured document images into structured, machine-readable output, is particularly challenging to evaluate. Models vary substantially in architecture, output format, and how they encode complex elements such as tables and formulas. These differences mean that the way a benchmark evaluation pipeline segments predictions, matches them to ground truth, and scores them can systematically advantage or disadvantage entire classes of systems.

Data distribution shift is a well recognised challenge for benchmark validity [30], and scoring metrics have also been widely scrutinised [22], though their effect on model rankings remains understudied in document parsing, where output heterogeneity makes such choices especially consequential. Alignment strategies, the procedures by which model predictions are matched to ground truth before scoring, are almost never compared, despite being the axis on which document parsing differs most from other evaluation settings.

Evaluation dataset, alignment strategy and scoring metric are varied independently, and the resulting leaderboards compared using Kendall’s $\tau$, top-$k$ churn, and pairwise reversals.

Figure 1. Evaluation dataset, alignment strategy and scoring metric are varied independently, and the resulting leaderboards compared using Kendall’s τ\tau, top-kk churn, and pairwise reversals.

OmniDocBench [25] has become the de facto benchmark for comparing document parsing systems, yet its own maintainers have identified matching-strategy bias, the penalising of correct content that is segmented differently from the annotation, as an important factor limiting its effectiveness [32]. Version 1.6 introduces a redesigned matching procedure in response. However, neither the bias nor the fix has been quantified, and their effect on model rankings remains unknown.

In this work, we present RankShift, a systematic study of ranking instability in document parsing evaluation. Using the OmniDocBench v1.5 dataset with the v1.6 evaluation protocol as a case study, we evaluate 16 OCR and vision-language models under controlled variation across three benchmark dimensions: alignment strategy, dataset composition, and scoring metric. For alignment, we compare six strategies formed by crossing three matching methods with two ground-truth formats: end-to-end (E2E), which scores predictions against OmniDocBench’s JSON annotations, and markdown-to-markdown (M2M), which scores them against the Markdown rendering of those annotations. For dataset composition, we compare rankings across the full benchmark, its nine page-type strata, five layout-type strata, and a scanned reconstruction of the same pages (Real5 [39]). For scoring, we compare three string based evaluation metrics: NED, BLEU, and METEOR.

Figure 1 summarizes how these benchmark design choices produce alternative model rankings and how RankShift measures their stability.

Our contributions are as follows:

  • We present the first study of ranking instability in document parsing, quantifying how model rankings shift under reasonable variation in evaluation design.

  • We identify the interaction between ground-truth format and matching method as an understudied source of ranking instability that, under segment-level matching, reverses nearly a third of all model pairs — rivalling dataset composition and substantially exceeding metric choice.

  • We propose a diagnostic procedure, based on a model’s profile of scores across alignment strategies, in which recurrent disagreement patterns suggest candidate failure modes such as hallucinated content or segmentation mismatch.

Related Work

Document Parsing Benchmarks

Appraising a parsed page fairly is difficult because multiple valid representations exist for the same underlying content. This has led to substantial divergence in how leading document parsing benchmarks score parsed outputs. olmOCR-Bench [27] and ParseBench [36] replace text-similarity scoring with binary unit testing, arguing that the former penalise trivial formatting differences [31] while missing failures that matter downstream [36]. Each test asserts a single property such as required text, reading order or table structure. In contrast, DOCR-Inspector [37] uses a vision-language model (VLM) as a judge to assess parsing quality across 28 error types. These approaches avoid whole-page matching of parsed outputs to reference annotations, and so avoid the difficulties of global alignment, while making each assertion or error label individually interpretable.

While these approaches mitigate the problems of global alignment, they introduce a closed-world assumption: text-similarity scoring accounts for every divergence between a prediction and its reference, whereas assertion- and judge-based approaches can only identify the failure modes they were designed to detect, so their coverage is fixed by design rather than by the output itself. DOCR-Inspector’s error taxonomy derives from manual analysis of a sample of documents and, though broad, is unlikely to capture all error types. Additionally multi-error examples form only 16% of its training data, synthesised from two to four non-interfering error types, whereas real-world errors are both more common and more likely to interact. The unit-test-based benchmarks inherit a similar selection bias: a test exists only where the generating model could read the content, so pages that defeat the generator yield fewer and easier assertions. OmniDocBench [25], by contrast, appraises output by matching full-page predictions to annotated ground truth and scoring the matched spans with text-similarity metrics. It remains the gold-standard benchmark against which most document parsing systems are compared.

OmniDocBench and its Ecosystem

OmniDocBench [25] is unique in the range of evaluation configurations it exposes: three matching strategies, two ground-truth formats, a choice of metrics, and attribute-level stratification. Its documentation justifies its default evaluation method only on the grounds that the JSON format preserves the category and attribute annotations that enable category-level filtering and attribute-stratified results, without discussing how any of these choices affects scores or rankings. The maintainers have since identified matching-strategy bias as a limitation, whereby matching predictions to annotations one-to-one penalises systems whose segment boundaries differ from the annotations’, even where the parsed content is correct. To address this, version 1.6 introduces Multi-Granularity Adaptive Matching (MGAM), which holds the ground truth fixed and searches for an optimal segmentation granularity on the prediction side [32]. The redesign is defined rather than evaluated, however: no comparison of scores or model rankings under the different matching strategies is reported, and it remains unclear which output characteristics each strategy systematically rewards or penalises, and why.

Several benchmarks inherit OmniDocBench’s evaluation methodology including InduOCRBench [29], MDPBench [19], MPDocBench-Parse [38] and Dr. DocBench [35]. Some retain both ground-truth formats without discussing the trade-off, while others adopt one silently. None appear to weigh the consequences of the alignment or metric choice, a decision that grows more consequential as the ecosystem widens.

Benchmark Sensitivity and Ranking Instability

Ranking instability under evaluation design variation is a recognised but understudied problem, though a growing body of work has begun to address it [10]. Post [26] and Marie et al. [21] highlight the difficulty of comparing even nominally identical scores such as BLEU across the literature, since implementation choices such as reference text preprocessing leave them effectively non-standardised. Similarly, Alzahrani et al. [2] showed that minor perturbations in popular multiple-choice question benchmarks, including changing the order of answers or the means of selecting them, can substantially alter LLM leaderboard rankings.

Benchmarks themselves have received similar scrutiny. The Benchmark Lottery [8] argues that rankings are fragile to benchmark task selection biases, which can unfairly favour models whose strengths align with the chosen tasks, at the expense of those that perform well on others. Benchmark² [28] attempts to quantify benchmark quality directly by evaluating benchmarks along three axes: cross-benchmark ranking consistency (how well the rankings a benchmark produces correlate with those of peer benchmarks), discriminability (how well it distinguishes between models), and capability alignment deviation (identifying problematic instances where strong models do worse than weaker models from the same family), and found significant variance in quality among widely used benchmarks. Attention has since turned to individual fields: SafetyRepro [17] found that on every AI safety alignment benchmark tested, configuration choice alone can flip which model is deemed safer, and Kendiukhov [15] found pairwise reversal rates from 0.0% to 32.1% across four evaluation protocol axes in gene regulatory network benchmarking.

Together, these studies establish that benchmark rankings are fragile to evaluation design. However, its impact in document parsing, which carries the distinctive difficulty of aligning model outputs to reference text, remains underexplored.

Methodology

Datasets

We evaluate all models on two datasets: the full OmniDocBench v1.5 benchmark and the Real5-OmniDocBench Scanning subset, which together support our three dataset configuration experiments.

OmniDocBench v1.5. OmniDocBench [25] is the primary benchmark used throughout this work. Version 1.5 comprises 1,355 bilingual PDF pages spanning nine document types; academic papers, textbooks, financial reports, research reports, slides, books, newspapers, exam papers, and handwritten notes, across five layout types. Pages are annotated at 19 block-level categories and 15 attribute labels, enabling end-to-end, module-level, and attribute-stratified evaluation.

Real5-OmniDocBench (Scanning). Real5-OmniDocBench [39] performs a full-scale, one-to-one physical reconstruction of all 1,355 OmniDocBench v1.5 pages across five real-world degradation conditions: Scanning, Warping, Screen-Photography, Illumination, and Skew. Only the Scanning condition, which represents the most common real-world document capture scenario, is used in the study. The one-to-one correspondence with OmniDocBench v1.5 is critical to our experimental design: because every scanned page shares its ground truth annotation with the digital counterpart, differences in model rankings between the two datasets can therefore be attributed to capture condition alone.

Models

We evaluate 16 models spanning specialist document and general-purpose VLMs Table 1.

Pipeline-based parsers. Docling [14] chains layout analysis, table recognition, and reading order prediction to produce structured markdown.

Specialist document VLMs. This group spans a range of scales and architectures: MinerU2.5 [23] (1.2B), GOT-OCR2 [33] (580M), GLM-OCR [9] (0.9B), PaddleOCR-VL-1.5 [6] (0.9B), Dolphin 1.5 [11] (0.5B), MonkeyOCR-Pro-3B [20] (3B), DeepSeek-OCR 2 [34] (3B), Chandra 2 [7] (5B), HunyuanOCR [13] (1B), Youtu-Parsing [3] (2.5B), dots.ocr [18] (1.7B), and RolmOCR [1] (8B), a fine-tuned Qwen2.5-VL-7B-Instruct variant. All are trained specifically for structured document output and produce markdown.

General-Purpose VLMs. We evaluate three closed-source frontier models via API using the inference scripts and prompts provided in the OmniDocBench repository: GPT-5.5 [24], Gemini 2.5 Pro [5], and Gemini 3.1 Flash [12]. These serve as general-purpose baselines, allowing us to assess whether ranking instability is specific to specialized document parsing models or reflects a property of the evaluation pipeline independent of model class.

A complete summary of model families, parameter scales, open-source availability, and output formats is provided in Table 1.

Table 1: Models evaluated in RankShift. Type: P = Pipeline, S-VLM = Specialist VLM, G-VLM = General-Purpose VLM. Output format indicates the primary structured output produced.

ModelTypeParamsOpenDoc-SpecialisedOutput
DoclingP–✓✓Markdown
MinerU2.5S-VLM1.2B✓✓Markdown
Chandra 2S-VLM5B✓✓Markdown
DeepSeek-OCR 2S-VLM3B✓✓Markdown
Dolphin 1.5S-VLM0.5B✓✓Markdown
dots.ocrS-VLM1.7B✓✓Markdown
GLMOCRS-VLM0.9B✓✓Markdown
GOT-OCR2.0S-VLM580M✓✓MD/LaTeX/HTML
Hunyuan OCRS-VLM1B✓✓Markdown
MonkeyOCR-pro-3BS-VLM3B✓✓Markdown
PaddleOCR-VL-1.5S-VLM0.9B✓✓Markdown
RolmOCRS-VLM8B1✓✓Markdown
Youtu-ParsingS-VLM2.5B✓✓Markdown
Gemini 2.5 ProG-VLM–××Markdown
Gemini 3.1 FlashG-VLM–××Markdown
GPT-5.5G-VLM–××Markdown

Table 1. Models evaluated in RankShift. Type: P = Pipeline, S-VLM = Specialist VLM, G-VLM = General-Purpose VLM. Output format indicates the primary structured output produced.

Experimental Setup

Experiment 1 — Dataset Dimension. We examine ranking stability under three types of dataset variation. First, page type: we partition OmniDocBench into its nine page-type strata (e.g. academic papers, newspapers, textbooks) and measure pairwise rank stability between every pair (92)=36\binom{9}{2} = 36 comparisons). Second, layout: we partition the same pages into five layout strata (e.g. single-column, double-column, three-column) and repeat the analysis ((52)=10\binom{5}{2} = 10 comparisons). Third, capture condition: we compare rankings on the original digital pages against the physically re-scanned Real5 pages. Strata are disjoint, so partition-based rankings are derived from entirely different pages, whereas the Real5 comparison holds page content fixed and varies only how the pages were captured. Table A1 in the appendix lists all configurations.

VariationConfigurationsComparisonsEvidence relationship
Page typePPT2PDF; academic literature; book; colorful textbook; exam paper; magazine; newspaper; note; research reportAll pairwise (36)Disjoint page subsets
Layout1+ column; double column; other layout; single column; three columnAll pairwise (10)Disjoint page subsets
Capture conditionOmniDocBench v1.5 digital pages; Real5 Scanning reconstructionsDigital vs. scanned (1)Paired versions of the same 1,355 pages

Table A1.

Experiment 2 — Alignment Dimension. We compare six alignment strategies, formed by crossing three matching methods with two ground-truth formats. In the four segment-level conditions, predictions are segmented on double line breaks before matching; the methods differ in how those segments are then matched to ground-truth segments: quick_match (adjacency-search-based truncation and merging of adjacent predicted segments, with hybrid formula–text matching to reduce scoring sensitivity to unicode formula output), simple_match (one-to-one assignment without truncation or merging), and no_split (whole-page concatenation into a single string, with no block-level alignment or attribute-level scoring). Each method is applied under two ground-truth formats: end-to-end (E2E), which scores against OmniDocBench’s JSON annotations, and markdown-to-markdown (M2M), which scores against their Markdown rendering. This yields (62)=15\binom{6}{2} = 15 pairwise comparisons.

The quick_match condition reported across all experiments uses the v1.6 implementation, which incorporates MGAM [32]. The redesign applies only to quick_match: its prediction-side granularity search presupposes that predicted segments can be merged or split to fit the ground truth, which simple_match and no_split exclude by definition. The simple_match and no_split procedures are therefore unchanged from v1.5.

Experiment 3 — Metric Dimension. Under a fixed alignment (quick_match E2E), we compare three scoring metrics: normalised edit distance (NED, reported as 1−NED1 - \mathrm{NED}), BLEU, and METEOR. All three are computed on the same aligned text segments, so rank differences reflect metric sensitivity rather than matching or pipeline choices.

Metric underlying reported rankings. Although OmniDocBench reports a composite score averaging text edit distance, table TEDS and formula CDM [31], all rankings analysed in the main text are derived from text edit distance alone, unless stated otherwise. This is because tables and formulas are extracted into separate categories before the text matcher runs [25]: the merging and splitting that distinguishes the alignment conditions acts only on the remaining text paragraphs. Reporting a composite would therefore dilute the manipulation.

Alignment Signatures

To characterise where alignment strategies disagree, we represent every complete page×model row by a six-character alignment signature. Page-level NED scores under the three E2E and three M2M alignments are discretised into zero, low, medium, or high buckets and concatenated in fixed order. Across 20,610 complete rows, this produces 301 observed signatures; the 20 most frequent cover 78.5% of rows (Figure B10).

Alignment signatures used in the Experiment 2 diagnostic procedure

Figure B10. Alignment signatures used in the Experiment 2 diagnostic procedure: the 20 most frequent patterns, covering 78.5% of complete page×model rows. Each row is one signature, with columns in the fixed order E2E quick_match (EqE_q), simple_match (EsE_s) and no_split (EnE_n), followed by the M2M counterparts MqM_q, MsM_s and MnM_n.

We combine signature frequency, score-cluster shape, corpus-level text statistics, and stratified manual review to associate recurrent E2E–M2M disagreement patterns with candidate failure mechanisms. Appendix C:2 provides the complete construction, thresholds, auxiliary labels, sensitivity analysis, and review protocol.

Stability Measures

We measure ranking instability between any two conditions AA and BB using three complementary statistics computed on the vector of per-model mean scores.

Kendall’s τ\tau measures global rank correlation (τ=1\tau= 1 indicates identical ordering; τ=0\tau= 0 is consistent with statistical independence).

Top-kk churn rate quantifies instability at the top of the leaderboard:

churn⁡(A,B,k)=k−∣Tk(A)∩Tk(B)∣k,(1)\operatorname{churn}(A, B, k) = \frac{k - \lvert T_k(A) \cap T_k(B) \rvert}{k}, \tag*{(1)}

where Tk(X)T_k(X) is the set of top-kk models under condition XX. A churn rate of 0 indicates the top-kk set is unchanged; 1 means complete turnover.

Pairwise reversal rate counts what fraction of model pairs {i,j}\{i, j\} have their relative ordering reversed between AA and BB:

rev⁡(A,B)=∣{(i,j):sgn⁡(siA−sjA)≠sgn⁡(siB−sjB)}∣∣{(i,j):siA≠sjA∧siB≠sjB}∣.(2)\operatorname{rev}(A, B) = \frac{\lvert\{(i,j) : \operatorname{sgn}(s_i^A - s_j^A) \ne\operatorname{sgn}(s_i^B - s_j^B)\}\rvert}{\lvert\{(i,j) : s_i^A \ne s_j^A \land s_i^B \ne s_j^B\}\rvert}. \tag*{(2)}

Bootstrap Confidence Intervals

All three statistics are accompanied by 95% bootstrap confidence intervals obtained from B=2,000B = 2{,}000 iterations. In each iteration, pages are resampled with replacement and per-model means are recomputed with multinomial weights. For conditions sharing the same page set (cross-dataset, Experiment 3), a single resample is applied to both conditions simultaneously, preserving page-level pairing. For conditions covering disjoint page sets (strata comparisons), the two strata are resampled independently, reflecting the true sampling uncertainty for each. For mean τ\tau across strata, bootstrap resamples are drawn at the model level.

Results

Ground-Truth Format Substantially Reorders Rankings Under Segment-Level Matching

Ranking stability within a single ground-truth format, across matching strategies. Varying only the matching method within a single ground-truth format produces moderate disagreement. Against the JSON annotations, τ\tau ranges from 0.550 to 0.767, with reversal rates of 11.7%–20.0%; against the Markdown rendering, agreement is comparable (τ=0.667\tau= 0.667–0.817), with reversal rates of 9.2%–16.7%. None of these six reversal rates are significant after FDR correction, indicating that they are consistent with ordinary variation within a fixed format (Table 2; Appendix Table A3).

DimensionComparisonKendall’s τ\tau [95% CI]Category
Alignmentquick_match vs simple_match (E2E)0.767 [0.575, 0.929]E2E vs E2E
Alignmentquick_match vs no_split (E2E)0.550 [0.229, 0.821]E2E vs E2E
Alignmentsimple_match vs no_split (E2E)0.583 [0.256, 0.860]E2E vs E2E
Alignmentquick_match vs no_split (M2M)0.817 [0.593, 0.965]M2M vs M2M
Alignmentsimple_match vs no_split (M2M)0.683 [0.449, 0.892]M2M vs M2M
Alignmentquick_match vs simple_match (M2M)0.667 [0.368, 0.911]M2M vs M2M
Alignmentquick_match (E2E) vs quick_match (M2M)0.400 [-0.037, 0.772]E2E vs M2M
Alignmentquick_match (E2E) vs simple_match (M2M)0.233 [-0.276, 0.699]E2E vs M2M
Alignmentquick_match (E2E) vs no_split (M2M)0.383 [-0.071, 0.776]E2E vs M2M
Alignmentsimple_match (E2E) vs quick_match (M2M)0.367 [-0.009, 0.682]E2E vs M2M
Alignmentsimple_match (E2E) vs simple_match (M2M)0.300 [-0.207, 0.741]E2E vs M2M
Alignmentsimple_match (E2E) vs no_split (M2M)0.383 [-0.037, 0.736]E2E vs M2M
Alignmentno_split (E2E) vs quick_match (M2M)0.717 [0.474, 0.912]E2E vs M2M
Alignmentno_split (E2E) vs simple_match (M2M)0.583 [0.274, 0.852]E2E vs M2M
Alignmentno_split (E2E) vs no_split (M2M)0.767 [0.504, 0.948]E2E vs M2M
DatasetOmniDocBench v1.5 vs Real5 (E2E quick_match NED)0.800 [0.589, 0.947]Cross-dataset
DatasetMean tau across page-type (data_source) strata, E2E quick_match0.475 [0.279, 0.646]Intra-dataset
DatasetMean tau across layout strata, E2E quick_match0.490 [0.247, 0.686]Intra-dataset
MetricNED vs BLEU (E2E quick_match)0.700 [0.463, 0.874]—
MetricNED vs METEOR (E2E quick_match)0.767 [0.486, 0.945]—
MetricBLEU vs METEOR (E2E quick_match)0.867 [0.720, 0.965]—

Table 2. Pairwise Kendall’s τ\tau between model rankings (mean page-level scores). 95% CIs from bootstrap resampling of models (n=16n = 16), B=2000B = 2000 replicates.

Ranking stability across ground-truth formats, within a single matching strategy. Three pairs hold the matching method fixed and vary only the ground-truth format, isolating its effect. Under no_split, where each page is scored as a single concatenated string, the two formats agree closely (τ=0.767\tau= 0.767, reversal rate 10.0%, not significant after FDR correction). Under the two segment-level methods, agreement collapses: τ=0.400\tau= 0.400 for quick_match and τ=0.300\tau= 0.300 for simple_match. Both formats are derived from the same annotations, and the divergence appears only when predictions are matched segment by segment. This is therefore the leaderboard-level consequence of the matching-strategy bias identified by the OmniDocBench authors [23].

Across the four pairs in which both conditions match at segment level, mean Kendall’s τ\tau is approximately 0.33 and the mean pairwise reversal rate is 31.7%. The most unstable pair, E2E quick_match versus M2M simple_match, reverses 35.8% of model pairs (95% CI [30.0%, 39.2%]) and has a top-5 churn rate of 0.60, meaning that three of the five models in the top set change. Table 2 summarises all pairwise τ\tau values, and Appendix C.3 reports the corresponding reversal-rate significance tests.

Diagnostic Protocol

The question of which evaluation protocol best agrees with human judgment of parsing quality remains open to investigation. Each appears to have distinct strengths and weaknesses that practitioners, benchmark creators, and OCR developers should treat explicitly. We therefore frame the E2E–M2M score gap as a candidate diagnostic signal, in which the direction and shape of disagreement may indicate which failure mode dominates.

On OmniDocBench v1.5, recurrent E2E-favouring families account for approximately 25% of complete page × model rows, compared with approximately 0.8% for the M2M-favouring families. Our manual review suggests that E2E is comparatively lenient on extra or loosely aligned text, whereas M2M can avoid some region-level equation, layout, or title-matching failures. The asymmetry may therefore reflect corpus composition rather than any general protocol superiority. Appendix C.3 reports the supporting signature clustering, mechanism summaries, manual-review protocol, and diagnostic tables.

Experiment 2 bump chart showing rank trajectories under alignment variation

Figure 2. Experiment 2 bump chart showing rank trajectories under alignment variation.

Dataset Composition Produces Instability Across Page-Type and Layout Strata

Page-type stratification. Splitting OmniDocBench by document type reveals substantially greater instability than the cross-dataset comparison. Under E2E quick_match NED, the mean agreement across all page-type pairs is τ=0.475\tau= 0.475, with individual comparisons ranging from τ=0.067\tau= 0.067 for academic literature versus research report to τ=0.717\tau= 0.717 for PPT2PDF versus magazine. Academic literature is a systematic outlier, producing the lowest agreement with nearly every other stratum; its near-zero correlation with research reports indicates that these two text-dense document types induce almost unrelated model orderings. This heterogeneity is obscured by aggregate results: dots.ocr ranks first overall but only tenth within the academic_literature stratum, a nine-position difference.

Layout-type stratification. Layout strata exhibit a similar degree of instability (mean τ=0.490\tau= 0.490, range [0.267, 0.600] under E2E quick_match NED). The least stable comparison is 1+ column versus three-column (τ=0.267\tau= 0.267), partly reflecting the small size of the three-column stratum (52 pages). The most stable pairs reach only τ=0.600\tau= 0.600, including 1+ column versus double column and other_layout versus three-column. Stratifying by layout can again reveal strengths the aggregate conceals: Dolphin 1.5 ranks 11th overall but fourth on three-column pages.

Cross-dataset (OmniDocBench vs. Real5 Scanning). For text-based NED, rankings are broadly preserved when moving from digital PDFs to scanned reconstructions (τ=0.800\tau= 0.800, 95% CI [0.589, 0.947]; pairwise reversal rate = 0.100, CI [0.067, 0.133]). However, there is still some movement. The top-5 set is not fully preserved (churn rate = 0.20): dots.ocr drops out and DeepSeek-OCR 2 enters. Additionally, five models shift by two or more positions: dots.ocr falls from 1st to 6th under Real5, MonkeyOCR-pro-3B moves from 12th to 14th, while PaddleOCR-VL-1.5, MinerU 2.5 and RolmOCR each improve by two ranks.

Metric Choice Is the Most Stable Dimension

Of the three evaluation dimensions, metric choice produces the least instability. All three pairwise comparisons yield τ≥0.700\tau\ge0.700: BLEU versus METEOR achieves the highest agreement (τ=0.867\tau= 0.867), followed by NED versus METEOR (τ=0.767\tau= 0.767) and NED versus BLEU (τ=0.700\tau= 0.700), giving a mean of τ≈0.78\tau\approx0.78.

Rank-displacement heatmap for Experiment 1 across layout strata

Figure 3. Rank-displacement heatmap for Experiment 1 across layout strata. Each cell shows how many leaderboard positions a model gains when ranked on B-pages instead of A-pages (rank A − rank B). +12+12 (blue) means the model moved 12 places toward #1; −3-3 (red) means it fell 3 places.

Despite strong overall correlation, individual models still change position under different metrics. dots.ocr leads under NED, whereas Youtu-Parsing leads under both BLEU and METEOR, rising three places from fourth, suggesting its outputs are rewarded more by n-gram overlap than by character-level edit distance. Hunyuan OCR moves from eighth under NED to fifth under BLEU, while MinerU 2.5 moves in the opposite direction, from fifth to seventh. Therefore even the most stable dimension tested produces measurable ranking instability, indicating that leaderboard positions are better read as approximate than as fixed.

Discussion

Leaderboard Position Is Highly Conditional

A model’s rank on a document parsing benchmark reflects not only its capability, but also the evaluation configuration under which it was assessed. The same predictions, scored under different alignment strategies, dataset compositions, or metrics, can produce substantially different orderings. A clear illustration of this is the reversal of rankings between GLMOCR and RolmOCR. Under E2E quick_match, GLMOCR ranks 2nd while RolmOCR ranks 14th. Under M2M simple_match, the positions invert, with GLMOCR falling to rank 13 and RolmOCR rising to rank 5. The direction of this reversal holds across every E2E/M2M pair including the no_split pair, where the two formats on average agree closely (Figure B9), so it reflects the ground-truth format rather than any particular matching method: GLMOCR’s output aligns more closely with OmniDocBench’s JSON annotations, while RolmOCR’s aligns more closely with their Markdown rendering. Applying the diagnostic protocol (Appendix § 9:§ 3) yields a potential explanation: GLMOCR produces substantially more formatting overhead, which region-based matching against the JSON annotations does not heavily penalise, whereas RolmOCR’s segment boundaries potentially diverge from the annotated blocks, which can cost it under JSON matching but not when the page is compared as Markdown. Neither ranking has been shown to be wrong; each reflects a different, equally defensible assumption about what constitutes correct output.

Focused bump chart comparing RolmOCR and GLMOCR across the six alignment settings in Experiment 2

Figure B9. Focused bump chart comparing RolmOCR and GLMOCR across the six alignment settings in Experiment 2. GLMOCR ranks higher under every E2E condition and RolmOCR under every M2M condition, reversing their relative order.

For practitioners, a profile of scores across reasonable design choices is therefore more informative than a single ordering, particularly when the differences can be read against candidate failure modes: it indicates which configurations a model performs well under, and so which document types and failure tolerances it suits.

Segment-Level Matching Amplifies the Effect of the Ground-Truth Format

The prediction segments are identical under both ground-truth formats; what differs is how the reference is divided. The Markdown rendering merges, omits and rewrites some annotated blocks, whereas the JSON format excludes eleven categories from the text score. Segment-to-segment matching has no slack for such differences, since a correctly parsed segment paired with the wrong reference can score near zero, whereas concatenating the page removes the pairing step entirely. This is the leaderboard-level consequence of the matching-strategy bias identified by the OmniDocBench authors [32].

MGAM targets precisely this step, searching for an optimal segmentation granularity on the prediction side while holding the ground truth fixed. Even with MGAM applied, however, the two ground-truth formats agree at only τ=0.400\tau= 0.400 under quick_match, so the choice of reference format remains consequential after the redesign.

Limitations and Future Work

Validation of the diagnostic protocol. The mapping from signature families to failure modes in § 4.2 is derived from score-signature clustering, aggregate text statistics, and targeted manual review of exemplar pages. It is not validated against a held-out labelled sample, so we report the mechanisms behind each family but not the precision with which a signature predicts the dominant failure mode on an unseen page. An annotation study, labelling a sample of pages with their dominant failure mode and measuring agreement with the predicted signature family, would turn the protocol from a descriptive mapping into an instrument with known error rates. Relatedly, our characterisation covers the most frequent families; the long tail of the signature vocabulary (301 signatures, of which the top 20 account for 78% of rows) remains unexamined.

Benchmark coverage. Our study is centred on OmniDocBench v1.5, which to our knowledge is the only document parsing benchmark exposing configurable alignment; benchmarks outside its ecosystem fix a single alignment procedure, or bypass it altogether, so the controlled comparison cannot currently be run elsewhere. Extending RankShift to inherit benchmarks that retain both alignment options (e.g. MDPBench, MPDocBench-Parse) would test whether the relative importance of alignment, dataset, and metric effects holds under different corpus compositions. This bears directly on our diagnostic protocol: the E2E-favouring signatures that dominate here reflect the document types present in OmniDocBench, and a math- or slide-heavy corpus should shift the balance toward the M2M-favouring families we observe only rarely. That the comparison is unavailable elsewhere is itself an argument for exposing alignment as an explicit, configurable variable.

Conclusion

Document parsing benchmarks are used to compare model performance, but rankings are conditional on evaluation design. RankShift shows that these design choices do not contribute equally: alignment strategy and dataset composition substantially reorder model rankings, while metric choice leaves them largely intact. Disagreement across alignment strategies is also potentially informative, since a model’s profile of scores may indicate which failure modes dominate its output. We therefore propose that benchmark releases report the evaluation configuration alongside results, together with rank-stability profiles under reasonable design choices.

Additional Tables

Supporting tables for the results reported in the main text are provided below.

Table A1: Dataset configurations used in Experiment 1. Partition-based comparisons use disjoint page subsets, while the capture-condition comparison pairs two versions of the same pages.

Table A2: Experiment 1 model rankings on OmniDocBench v1.5 vs. Real5 Scanning (n=16n = 16 models, E2E quick_match text NED). Δ\Delta = OmniDocBench rank − Real5 rank (1 = best); negative Δ\Delta means a worse rank under Real5.

ModelOmniDocBench rankReal5 rankΔ\Delta Real5 position
dots.ocr16−5
GLMOCR220
PaddleOCR-VL-1.531+2
Youtu-Parsing440
MinerU2.553+2
DeepSeek-OCR 265+1
Chandra 2770
Hunyuan OCR89−1
Gemini 3.1 Flash98+1
Gemini 2.5 Pro1011−1
Dolphin 1.51110+1
MonkeyOCR-pro-3B1214−2
GPT-5.513130
RolmOCR1412+2
Docling15150
GOT-OCR2.016160

Table A2.

Statistical Significance of Rank Reversals

For Experiment 2 we report formal significance testing at three complementary levels, each with Benjamini–Hochberg FDR correction at α=0.05\alpha= 0.05.

Table A3: Alignment-pair rank reversal rates (n=16n = 16 models, 120 model pairs per alignment pair). One-sided binomial tests vs. within-E2E baseline (∼16.7%\sim16.7\%); qq-values from Benjamini–Hochberg FDR over 15 alignment pairs (α=0.05\alpha= 0.05). 95% CIs from page bootstrap. An asterisk indicates significance after FDR correction (q<0.05q < 0.05).

ComparisonRate95% CIqq-value
Cross-paradigm (E2E ↔ M2M)
E2E quick_match vs. M2M simple_match35.8%[30.0%, 39.2%]4.8×10−64.8 \times10^{-6}*
E2E simple_match vs. M2M simple_match35.0%[27.5%, 36.7%]6.8×10−66.8 \times10^{-6}*
E2E simple_match vs. M2M quick_match30.0%[23.3%, 32.5%]0.001*
E2E simple_match vs. M2M no_split29.2%[21.7%, 31.7%]0.002*
E2E quick_match vs. M2M no_split28.3%[21.7%, 31.7%]0.003*
E2E quick_match vs. M2M quick_match25.8%[20.8%, 29.2%]0.018*
E2E no_split vs. M2M simple_match20.0%[15.8%, 26.7%]0.363
E2E no_split vs. M2M quick_match12.5%[8.3%, 17.5%]0.986
E2E no_split vs. M2M no_split10.0%[5.8%, 15.8%]0.986
Within E2E
E2E quick_match vs. E2E simple_match11.7%[7.5%, 14.2%]0.986
E2E quick_match vs. E2E no_split20.0%[15.0%, 25.0%]0.363
E2E simple_match vs. E2E no_split18.3%[14.2%, 23.3%]0.475
Within M2M
M2M quick_match vs. M2M simple_match16.7%[15.8%, 24.2%]0.372
M2M quick_match vs. M2M no_split9.2%[6.7%, 14.2%]0.986
M2M simple_match vs. M2M no_split15.8%[13.3%, 20.8%]0.459

Table A3.

Alignment-pair reversal rate. For each pair of alignments, we count how often two models’ relative order on a page reverses between the alignments (120 model pairs per alignment pair for 16 models) and test whether cross-paradigm reversal rates exceed the mean within-E2E baseline (∼16.7%\sim16.7\%), which captures normal ranking noise under a fixed protocol. We use one-sided binomial tests with BH-FDR correction over the 15 alignment pairs. Six of the nine cross-paradigm (E2E↔M2M) pairs show significantly elevated reversal rates (25–36% vs. the ∼17%\sim17\% within-protocol baseline); none of the six within-paradigm pairs do. Rank instability is therefore protocol-specific rather than general noise. All three nonsignificant cross-paradigm comparisons involve E2E no_split; the lowest rate, E2E no_split vs. M2M no_split (10%), is expected because both alignments use whole-page matching, so the paradigm distinction largely collapses.

Per-model rank displacement. Page-bootstrap tests (2,000 resamples) confirm that individual models shift rank systematically across alignments: 78 model×\timesalignment-pair displacements are FDR-significant, including 51 under cross E2E↔M2M pairs. For example, RolmOCR improves by eleven ranks under M2M no_split vs. E2E quick_match, whereas GLMOCR worsens by eight ranks under the same pair.

Model-pair page flips. At page granularity, 869 of 1,800 model-pair×\timesalignment-pair combinations show significant head-to-head flip rates under two-sided binomial tests against the pair-mean baseline.

Together, these tests establish that the reversals are systematic and statistically robust rather than artifacts of indistinguishable neighbouring models.

Additional Figures

This appendix presents supplementary figures for the three instability experiments.

Rank displacement: OmniDoc vs Real5 (Experiment 1)

Figure B1. Cross-dataset rank displacement from OmniDocBench v1.5 to Real5 Scanning in Experiment 1. Values report OmniDocBench rank minus Real5 rank: positive (blue) values indicate movement toward rank 1 on Real5, negative (red) values indicate a lower rank on Real5, and zero indicates no change.

Pairwise reversal-rate heatmap for Experiment 1

Figure B2. Pairwise reversal-rate heatmap for Experiment 1. Each cell gives the share of the 120 model pairs whose ordering reverses between two of the nine OmniDocBench page-type strata (36 comparisons).

Full rank heatmap for Experiment 1 showing model ranks within each of nine data source strata

Figure B3. Full rank heatmap for Experiment 1: each cell represents a model’s rank within each of the nine OmniDocBench page-type strata, where 1 is the best-scoring model on that page type and 16 the worst.

Full rank heatmap for Experiment 1

Figure B4. Full rank heatmap for Experiment 1: each cell gives a model’s rank within one of the five OmniDocBench layout strata, where 1 is the best-scoring model on that layout type and 16 the worst.

Top-1 churn across Experiment 1 comparison settings

Figure B5. Top-1 churn across Experiment 1 comparison settings. Each cell indicates whether the highest-scoring model differs between two page-type strata: 1.00 for a different top-ranked model, 0.00 for the same.

Top-3 churn across Experiment 1 comparison settings

Figure B6. Top-3 churn across Experiment 1 comparison settings. Each cell gives the proportion of the top-three set that changes between two page-type strata, from 1.00 where none of the three models are retained to 0.00 where all top-three models are the same.

Top-5 churn across Experiment 1 comparison settings

Figure B7. Top-5 churn across Experiment 1 comparison settings. Each cell gives the proportion of the top-five set that changes between two page-type strata, ranging from 0.20, where one of the five models differs, to 0.80, where four differ.

Cross-dataset rank scatter for Experiment 1

Figure B8. Cross-dataset rank scatter for Experiment 1, comparing model ranks on OmniDocBench v1.5 with the Real5 Scanning subset.

Model Ranking By Evaluation Metric (Experiment 3)

Figure B11. Experiment 3 bump chart showing each model’s rank trajectory across the three scoring metrics: NED, BLEU and METEOR. Rankings are largely preserved, though the top-ranked model changes, with dots.ocr leading under NED and Youtu-Parsing under both BLEU and METEOR.

Kendall τ Between Evaluation Metrics (Per-Model Mean Scores, Experiment 3)

Figure B12. Heatmap of pairwise rank agreement across the three Experiment 3 scoring metrics. All three pairs show high agreement, with Kendall’s τ\tau between 0.700 and 0.867.

Detailed Methodology

This appendix provides additional methodological details and sensitivity analyses supporting the main and supplemental experiments.

Implementation Details

All experiments were run on NVIDIA H200 GPUs. Inference followed the official OmniDocBench evaluation scripts where available; open-source VLMs were served using vLLM [16] where supported.

API models. GPT-5.5, Gemini 2.5 Pro, and Gemini 3.1 Flash were queried using OmniDocBench’s reference inference scripts, with the benchmark’s fixed system prompt instructing the model to output page content in Markdown. Only the output token cap was set; temperature and sampling parameters were left at provider defaults.

Prompt standardisation. All VLMs received the same page image and a standardised prompt requesting structured markdown output. No model-specific prompt engineering was applied beyond each model’s official inference script in OmniDocBench; where no official script existed, the prompt followed the conventions of the closest comparable model.

Alignment Signature Construction

Unit of analysis and scores. For every model mm and page pp, we collect the page-level NED score (1−1 - normalised edit distance) under each of the six Experiment 2 alignments. We retain only rows with a valid score under all six alignments (N=20,610N = 20{,}610 page×model rows for 16 models).

Discretisation. Each score is mapped to one of four buckets:

BucketNED rangeLabelInterpretation
0≤0.01\le0.01zeroEmpty or collapsed match
L(0.01,0.40)(0.01, 0.40)lowPoor page-level match
M[0.40,0.85)[0.40, 0.85)mediumPartial match
H≥0.85\ge0.85highStrong match

Table 1 (PDF p. 27).

Cutoffs were chosen a priori for interpretability rather than from corpus quantiles. Under the pooled 16-model distribution they yield bucket fractions 0: 2.6%, L: 8.0%, M: 27.2%, and H: 62.2%. Alignment-specific distributions differ markedly: E2E quick_match is heavily right-skewed (median NED = 0.993; 86.7% H), whereas M2M simple_match is more dispersed (median = 0.836; 48.0% H and 34.2% M). Fixed thresholds therefore encode a common absolute notion of “good page match” rather than per-alignment relative standing.

We did not rerun the full signature analysis under alternative cutoffs. Qualitatively, the conclusions appear robust to modest boundary shifts, since cross-paradigm gaps are large relative to bucket width on the affected rows, and the top-20 signatures and E2E-high/M2M-low families persist under ±0.05 perturbations of the L/M boundary in spot checks. Perference-based bucketing would increase the observed signature vocabulary without necessarily changing the dominant families.

Signature construction. Buckets are concatenated in the fixed order

Eq∥Es∥En∥Mq∥Ms∥MnE_q \mathbin{\|} E_s \mathbin{\|} E_n \mathbin{\|} M_q \mathbin{\|} M_s \mathbin{\|} M_n

The first three positions denote E2E quick_match, simple_match, and no_split; the final three denote the corresponding M2M methods. For example, HHMHHH denotes high scores under every condition except E2E no_split, while HHHMHO denotes a collapsed M2M no_split slot. The vocabulary has at most 46=4,0964^6 = 4{,}096 patterns, of which 301 are observed. Bucketing the NED scores groups rows into recurring types of case, such as those scoring high under every alignment or high under E2E and low under M2M. These groups can be ranked by frequency and inspected as a set, which continuous scores would make impractical since exact repeats are rare.

Derived structure. Each row also receives a score cluster shape, describing how raw scores group when rounded to three decimals (e.g. identical, E2E ≠\ne M2M, E2E split, M2M split, or loner), and a uniformity tier that distinguishes numerically identical from alignment-sensitive rows. At page level, we compute modal-signature fraction and normalised Shannon entropy. We flag mechanism-goldmine pages where at least 65% of models share one signature, entropy is at most 0.55, and a consensus bad slot (0 or L) appears in at least 50% of model rows.

Failure-mode analysis. We proceed in four stages:

  1. Rank exact signatures by frequency and test model enrichment in each top pattern using one-sided binomial tests with Benjamini–Hochberg FDR correction. The top 20 signatures cover 78.5% of rows (Figure B10).

  1. Assign rule-based tags (e.g., all_high and mn_zero) and group cross-paradigm families by the E2E–M2M gap direction.

  1. Compare family-level prediction length, format-character ratio, header density, and M2M insertion dominance with corpus baselines and test model-level correlations.

  1. Review up to five distinct pages per (top-20 signature, cluster-shape) cell (∼\sim 490 exemplars), plus models diverging from the modal signature on approximately 19 mechanism-goldmine pages. Reviewers check for excess or hallucinated content, formatting overhead, region-collapse artefacts, and layout mis-pairing.

Diagnostic Protocol Details

The diagnostic reading combines automatic alignment-signature clustering, cross-alignment mechanism summaries, and targeted manual review. Tables C1 and C2 summarise frequent recurring families on OmniDocBench v1.5. Frequencies are the share of listed exemplar signatures in each row; the rows are illustrative families rather than an exhaustive partition.

E2E-favouring signatures listed in Table C1 account for approximately 25% of page×\timesmodel rows (12% excess-content and 13% format-overhead families), compared with approximately 0.8% for the listed M2M-favouring signatures. The broader pattern “E2E all-high, M2M not all-high” covers approximately 17% of rows, whereas M2M-favouring rows with all-E2E 0/L slots account for approximately 0.2%.

Observed patternLikely causeWhat to checkExample signaturesFreq.Recommendation
E2E high, M2M lowerExcess content (hallucinations, captions, duplication)Pred. length vs. GT; extra spansHHMMM, HHHMMM, HHMMML, HHHHMM, HHHMML, HHLLL, HHMML∼\sim 12%Report both protocols; inspect extra output when E2E ≫\gg M2M
E2E high, M2M low/medium, heavy syntaxFormatting overhead (Markdown/HTML penalised byte-wise)Headers/HTML; stripped vs. full MarkdownHHHHHM, HHHMHM, HHMMLM, HHHHLH, HLHLHH, HHHMLM∼\sim 13%Pair region-based scores with M2M; report syntax-stripped variants

Table C1. Diagnostic families where E2E >> M2M (OmniDocBench v1.5, 16 models; N=20,610N = 20{,}610 page×\timesmodel rows). Frequencies are the combined share of listed signatures.

This frequency gap does not establish that E2E is globally superior. OmniDocBench is dominated by cases where region-based matching is lenient on extra or loosely aligned text and full-page Markdown comparison is stricter. Math-heavy benchmarks may produce more M2M>>E2E equation-mispairing cases such as LLLHHH or LLMHMM; title-heavy slides or worksheets may produce more OO*×\timesHHH-type region-collapse cases.

Observed patternLikely causeWhat to checkExample signaturesFreq.Recommendation
E2E low, M2M high (math)Equation/block mis-pair under E2EEquation blocks in GTLLLHHH, LLLHMM, LLMHMM∼0.02%\sim0.02\%Stratify math-heavy pages; audit block matching
E2E low, M2M high (layout)Column/reading-order break under concatenationBlock order vs. M2M concatenationMMMHHH, MMMHMH, MMLHMH, HLMHLH, LLLMML, 00LLLL∼0.8%\sim0.8\%Report M2M on layout-heavy pages
E2E title/box slot empty, M2M body correctRegion-slot failure (title dropped)GT title region vs. E2E match00MHHH, 00HHHH, 00MHHM∼0.03%\sim0.03\%Inspect region-level matching, not page OCR quality

Table C2. Diagnostic families where M2M > E2E (same corpus; rare on OmniDocBench v1.5).

References

  1. [1]Reducto AI. Rolmocr: A faster, lighter open source ocr model. Hugging Face model card, 2025.
  2. [2]Norah A. Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Z. Alsubaie, Yousef Almushayqih, Faisal Abdulrahman Mirza, Nouf M. Alotaibi, Nora Al-Twaire sh, Areeb Alowisheq, M. Saiful Bari, and Haidar Khan. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781, 2024. doi: 10.48550/arXiv.2402.01781.DOI
  3. [3]Haoyu Cao et al. Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding. arXiv preprint arXiv:2601.20430, 2026. URL https://arxiv.org/abs/2601.20430.
  4. [4]Center for AI Safety, Scale AI, and HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature, 649:1139–1146, 2026. doi: 10.1038/s41586-025-09962-4.
  5. [5]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. doi: 10.48550/arXiv.2507.06261.arxiv.org/abs/2507.06261
  6. [6]Cheng Cui et al. PaddleOCR-VL-1.5: Towards a multi-task 0.9b VLM for robust in-the-wild document parsing. arXiv preprint arXiv:2601.21957, 2026.arxiv.org/abs/2601.21957
  7. [7]Datalab. Chandra ocr 2. Hugging Face model card, 2025.
  8. [8]Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107.07002, 2021. doi: 10.48550/arXiv.2107.07002.DOI
  9. [9]Shuaiqi Duan et al. GLM-OCR technical report. arXiv preprint arXiv:2603.10910, 2026.arxiv.org/abs/2603.10910
  10. [10]Maria Eriksson, Erasmo Purificato, Arman Noroozian, João Vinagre, Guillaume Chaslot, Emilia G’omez, and David Fernandez-Llorca. Can we trust AI benchmarks? an interdisciplinary review of current issues in AI evaluation. arXiv preprint arXiv:2502.06559, 2025.
  11. [11]Hao Feng, Shu Wei, Xiang Fei, Wei Shi, Yingdong Han, Lei Liao, Jinghui Lu, Binghong Wu, Qi Liu, Chunhui Lin, Jingqun Tang, Hao Liu, and Can Huang. Dolphin: Document image parsing via heterogeneous anchor prompting. arXiv preprint arXiv:2505.14059, 2025.arxiv.org/abs/2505.14059
  12. [12]Google. Gemini 3.1 flash live preview. Google AI for Developers model documentation, 2026. URL https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview. Accessed 2026-08-29.
  13. [13]Hunyuan Vision Team et al. HunyuanOCR Technical Report. arXiv preprint arXiv:2511.19575, 2025. URL https://arxiv.org/abs/2511.19575.
  14. [14]IBM Research and Contributors. Docling: Technical report. GitHub repository and technical report, 2024.
  15. [15]Ihor Kendiukhov. Quantifying ranking instability across evaluation protocol axes in gene regulatory network benchmarking. arXiv preprint arXiv:2603.03493, 2026. doi: 10.48550/arXiv.2603.03493.DOI
  16. [16]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. arXiv preprint arXiv:2309.06180, 2023. doi: 10.48550/arXiv.2309.06180.arxiv.org/abs/2309.06180
  17. [17]Yanhang Li, Zhichao Fan, and Zexin Zhuang. SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks. arXiv preprint arXiv:2605.25492, 2026. doi: 10.48550/arXiv.2605.25492.DOI
  18. [18]Yumeng Li, Guang Yang, Hao Liu, Bowen Wang, and Colin Zhang. dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model. arXiv preprint arXiv:2512.02498, 2025. URL https://arxiv.org/abs/2512.02498.
  19. [19]Zhang Li, Zhibo Lin, Qiang Liu, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiajun Song, Jiarui Zhang, Xiang Bai, and Yu liang Liu. MDPBench: A benchmark for multilingual document parsing in real-world scenarios. arXiv preprint arXiv:2603.28130, 2026.arxiv.org/abs/2603.28130
  20. [20]Zhang Li et al. Monkeyocr: Document parsing with a structure-recognition-relation triplet paradigm. arXiv preprint arXiv:2506.05218, 2025.arxiv.org/abs/2506.05218
  21. [21]Benjamin Marie, Atsushi Fujita, and Raphael Rubino. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. arXiv preprint arXiv:2106.15195, 2021. doi: 10.48550/arXiv.2106.15195.arxiv.org/abs/2106.15195
  22. [22]Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, 2020.arxiv.org/abs/2006.06264
  23. [23]Junbo Niu et al. MinerU2.5: A decoupled vision-language model for efficient high-resolution document parsing. arXiv preprint arXiv:2509.22186, 2025. URL https://arxiv.org/abs/2509.22186.
  24. [24]OpenAI. GPT-5.5 model documentation. OpenAI developer documentation, 2026.
  25. [25]Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He. OmniDocBench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2412.07626
  26. [26]Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, 2018.arxiv.org/abs/1804.08771
  27. [27]Jake Poznanski, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. olmocr: Unlocking trillions of tokens in pdfs with vision language models. arXiv preprint arXiv:2502.18443, 2025.arxiv.org/abs/2502.18443
  28. [28]Qi Qian, Chengsong Huang, Jingwen Xu, Changze Lv, Muling Wu, Wenhao Liu, Xiaohua Wang, Zhenghua Wang, Zisu Huang, Muzhao Tian, Jianhan Xu, Kun Hu, He-Da Wang, Yao Hu, Xuanjing Huang, and Xiaoqing Zheng. Benchmark² : Systematic evaluation of llm benchmarks. arXiv preprint arXiv:2601.03986, 2026.arxiv.org/abs/2601.03986
  29. [29]Lin Sun, Dexian Wang, Jingang Huang, Linglin Zhang, Change Jia, Zhengwei Cheng, and Xiangzheng Zhang. When good OCR is not enough: Benchmarking OCR robustness for retrieval-augmented generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 884–894. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.acl-industry.60.DOI
  30. [30]Rohan Taori, Achal Dave, Vaishaal Shankar, et al. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2007.00644
  31. [31]Bin Wang, Fan Wu, Linke Ouyang, Zhuangcheng Gu, Rui Zhang, Renqiu Xia, Botian Shi, Bo Zhang, and Conghui He. Image over text: Transforming formula recognition evaluation with character detection matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19681–19690, June 2025. doi: 10.1109/CVPR52734.2025.01833. URL https://ieeexplore.ieee.org/document/11094880.DOI
  32. [32]Bin Wang et al. MinerU2.5-Pro: Pushing the limits of data-centric document parsing at scale. arXiv preprint arXiv:2604.04771, 2026. doi: 10.48550/arXiv.2604.04771.DOI
  33. [33]Haoran Wei et al. General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model. arXiv preprint arXiv:2409.01704, 2024.arxiv.org/abs/2409.01704
  34. [34]Haoran Wei et al. DeepSeek-OCR 2: Visual causal flow. arXiv preprint arXiv:2601.20552, 2026.arxiv.org/abs/2601.20552
  35. [35]Minglai Yang, Xinyan Velocity Yu, Pengyuan Li, Xinyu Guo, Zhenting Qi, Konwoo Kim, Longtian Ye, Xiaolong Luo, Jinhbe Li, Henry Zhang, Haris Riaz, Xuan Zhang, Yunze Xiao, Bangya Liu, Tom Tang, Yunfei Zhao, Qunshu Lin, Zihan Wang, Minghao Liu, Michael Lingzhi Li, Yilun Du, Jesse Thomason, Rogerio Feris, Alex Pentland, and Zexue He. Dr. DocBench: A comprehensive benchmark for expert-level and difficult document parsing. arXiv preprint arXiv:2606.01393, 2026.arxiv.org/abs/2606.01393
  36. [36]Boyang Zhang, Sebastián G. Acosta, Preston Carlson, Sacha Bron, Pierre-Loïc Doulcet, Daniel B. Ospina, and Simon Suo. ParseBench: A document parsing benchmark for ai agents. arXiv preprint arXiv:2604.08538, 2026. doi: 10.48550/arXiv.2604.08538.DOI
  37. [37]Qintong Zhang et al. DOCR-Inspector: Fine-grained and automated evaluation of document parsing with VLM. arXiv preprint arXiv:2512.10619, 2025.arxiv.org/abs/2512.10619
  38. [38]Bangbang Zhou, Hangdi Xing, Yifan Chen, Jianjun Xu, Qi Zheng, Feiyu Gao, Zhibo Yang, Shuai Bai, Ming Yan, Jieping Ye, and Hongtao Xie. MPDocBench-Parse: Benchmarking practical multi-page document parsing. arXiv preprint arXiv:2605.22100, 2026.arxiv.org/abs/2605.22100
  39. [39]Changda Zhou et al. Real5-OmniDocBench: A full-scale physical reconstruction benchmark for robust document parsing in the wild. arXiv preprint arXiv:2603.04205, 2026.arxiv.org/abs/2603.04205

Paper details

vlnc:W_z8hey2ne
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents