WorksComputer Science
RankShift: Evaluating Ranking Instability Across Benchmark Dimensions in Document Parsing
Abstract
Document parsing benchmarks are often treated as stable measures of model quality, yet rankings may depend substantially on benchmark design choices. In this work, we present RankShift, a systematic study comparing OCR and vision-language models (VLMs) under controlled variation in dataset composition, matching strategy, and scoring methodology. Using OmniDocBench as a case study, we evaluate the ranking stability of 16 models across 3 dataset configurations, 6 matching strategies, and 3 scoring metrics. We quantify rank stability using pairwise rank correlation, top-k churn, and pairwise reversal rate. Our results show that the interaction between matching strategy and ground truth format is a substantial but understudied driver of instability: scoring against OmniDocBench's JSON annotations versus its Markdown rendering yields a mean Kendall's tau of approximately 0.33 under region-level matching strategies, with individual pairs as low as tau = 0.23, compared with tau = 0.77 when each page is scored as a single concatenated string. Dataset composition produces comparable instability across page-type strata (mean tau = 0.48) and layout-type strata (mean tau = 0.49), while the cross-dataset shift from OmniDocBench to Real5's Scanning reconstruction of the same pages is more stable (tau = 0.80). Metric choice is the most stable dimension, with mean tau of approximately 0.78 across NED, BLEU, and METEOR. These findings indicate that leaderboard position is not solely a property of model capability, motivating leaderboards that report not only headline scores but also the robustness of rankings across reasonable design choices.
Introduction
Benchmarks are essential tools for tracking progress in machine learning, yet they are often treated as static and objective arbiters of model capability [4, 28, 10]. In practice, benchmarks encode a range of design choices, including dataset composition, pre- and post-processing steps, and scoring methodology. The effects of these decisions on model rankings are rarely made explicit and remain largely understudied in document parsing. As benchmarks become increasingly saturated and susceptible to overfitting, these choices take on greater significance: small differences in evaluation design can determine a model’s overall ranking, independent of genuine capability differences.
Document parsing, the task of converting unstructured document images into structured, machine-readable output, is particularly challenging to evaluate. Models vary substantially in architecture, output format, and how they encode complex elements such as tables and formulas. These differences mean that the way a benchmark evaluation pipeline segments predictions, matches them to ground truth, and scores them can systematically advantage or disadvantage entire classes of systems.
Data distribution shift is a well recognised challenge for benchmark validity [30], and scoring metrics have also been widely scrutinised [22], though their effect on model rankings remains understudied in document parsing, where output heterogeneity makes such choices especially consequential. Alignment strategies, the procedures by which model predictions are matched to ground truth before scoring, are almost never compared, despite being the axis on which document parsing differs most from other evaluation settings.

Figure 1. Evaluation dataset, alignment strategy and scoring metric are varied independently, and the resulting leaderboards compared using Kendall’s , top- churn, and pairwise reversals.
OmniDocBench [25] has become the de facto benchmark for comparing document parsing systems, yet its own maintainers have identified matching-strategy bias, the penalising of correct content that is segmented differently from the annotation, as an important factor limiting its effectiveness [32]. Version 1.6 introduces a redesigned matching procedure in response. However, neither the bias nor the fix has been quantified, and their effect on model rankings remains unknown.
In this work, we present RankShift, a systematic study of ranking instability in document parsing evaluation. Using the OmniDocBench v1.5 dataset with the v1.6 evaluation protocol as a case study, we evaluate 16 OCR and vision-language models under controlled variation across three benchmark dimensions: alignment strategy, dataset composition, and scoring metric. For alignment, we compare six strategies formed by crossing three matching methods with two ground-truth formats: end-to-end (E2E), which scores predictions against OmniDocBench’s JSON annotations, and markdown-to-markdown (M2M), which scores them against the Markdown rendering of those annotations. For dataset composition, we compare rankings across the full benchmark, its nine page-type strata, five layout-type strata, and a scanned reconstruction of the same pages (Real5 [39]). For scoring, we compare three string based evaluation metrics: NED, BLEU, and METEOR.
Figure 1 summarizes how these benchmark design choices produce alternative model rankings and how RankShift measures their stability.
Our contributions are as follows:
We present the first study of ranking instability in document parsing, quantifying how model rankings shift under reasonable variation in evaluation design.
We identify the interaction between ground-truth format and matching method as an understudied source of ranking instability that, under segment-level matching, reverses nearly a third of all model pairs — rivalling dataset composition and substantially exceeding metric choice.
We propose a diagnostic procedure, based on a model’s profile of scores across alignment strategies, in which recurrent disagreement patterns suggest candidate failure modes such as hallucinated content or segmentation mismatch.
Related Work
Document Parsing Benchmarks
Appraising a parsed page fairly is difficult because multiple valid representations exist for the same underlying content. This has led to substantial divergence in how leading document parsing benchmarks score parsed outputs. olmOCR-Bench [27] and ParseBench [36] replace text-similarity scoring with binary unit testing, arguing that the former penalise trivial formatting differences [31] while missing failures that matter downstream [36]. Each test asserts a single property such as required text, reading order or table structure. In contrast, DOCR-Inspector [37] uses a vision-language model (VLM) as a judge to assess parsing quality across 28 error types. These approaches avoid whole-page matching of parsed outputs to reference annotations, and so avoid the difficulties of global alignment, while making each assertion or error label individually interpretable.
While these approaches mitigate the problems of global alignment, they introduce a closed-world assumption: text-similarity scoring accounts for every divergence between a prediction and its reference, whereas assertion- and judge-based approaches can only identify the failure modes they were designed to detect, so their coverage is fixed by design rather than by the output itself. DOCR-Inspector’s error taxonomy derives from manual analysis of a sample of documents and, though broad, is unlikely to capture all error types. Additionally multi-error examples form only 16% of its training data, synthesised from two to four non-interfering error types, whereas real-world errors are both more common and more likely to interact. The unit-test-based benchmarks inherit a similar selection bias: a test exists only where the generating model could read the content, so pages that defeat the generator yield fewer and easier assertions. OmniDocBench [25], by contrast, appraises output by matching full-page predictions to annotated ground truth and scoring the matched spans with text-similarity metrics. It remains the gold-standard benchmark against which most document parsing systems are compared.
OmniDocBench and its Ecosystem
OmniDocBench [25] is unique in the range of evaluation configurations it exposes: three matching strategies, two ground-truth formats, a choice of metrics, and attribute-level stratification. Its documentation justifies its default evaluation method only on the grounds that the JSON format preserves the category and attribute annotations that enable category-level filtering and attribute-stratified results, without discussing how any of these choices affects scores or rankings. The maintainers have since identified matching-strategy bias as a limitation, whereby matching predictions to annotations one-to-one penalises systems whose segment boundaries differ from the annotations’, even where the parsed content is correct. To address this, version 1.6 introduces Multi-Granularity Adaptive Matching (MGAM), which holds the ground truth fixed and searches for an optimal segmentation granularity on the prediction side [32]. The redesign is defined rather than evaluated, however: no comparison of scores or model rankings under the different matching strategies is reported, and it remains unclear which output characteristics each strategy systematically rewards or penalises, and why.
Several benchmarks inherit OmniDocBench’s evaluation methodology including InduOCRBench [29], MDPBench [19], MPDocBench-Parse [38] and Dr. DocBench [35]. Some retain both ground-truth formats without discussing the trade-off, while others adopt one silently. None appear to weigh the consequences of the alignment or metric choice, a decision that grows more consequential as the ecosystem widens.
Benchmark Sensitivity and Ranking Instability
Ranking instability under evaluation design variation is a recognised but understudied problem, though a growing body of work has begun to address it [10]. Post [26] and Marie et al. [21] highlight the difficulty of comparing even nominally identical scores such as BLEU across the literature, since implementation choices such as reference text preprocessing leave them effectively non-standardised. Similarly, Alzahrani et al. [2] showed that minor perturbations in popular multiple-choice question benchmarks, including changing the order of answers or the means of selecting them, can substantially alter LLM leaderboard rankings.
Benchmarks themselves have received similar scrutiny. The Benchmark Lottery [8] argues that rankings are fragile to benchmark task selection biases, which can unfairly favour models whose strengths align with the chosen tasks, at the expense of those that perform well on others. Benchmark² [28] attempts to quantify benchmark quality directly by evaluating benchmarks along three axes: cross-benchmark ranking consistency (how well the rankings a benchmark produces correlate with those of peer benchmarks), discriminability (how well it distinguishes between models), and capability alignment deviation (identifying problematic instances where strong models do worse than weaker models from the same family), and found significant variance in quality among widely used benchmarks. Attention has since turned to individual fields: SafetyRepro [17] found that on every AI safety alignment benchmark tested, configuration choice alone can flip which model is deemed safer, and Kendiukhov [15] found pairwise reversal rates from 0.0% to 32.1% across four evaluation protocol axes in gene regulatory network benchmarking.
Together, these studies establish that benchmark rankings are fragile to evaluation design. However, its impact in document parsing, which carries the distinctive difficulty of aligning model outputs to reference text, remains underexplored.
Methodology
Datasets
We evaluate all models on two datasets: the full OmniDocBench v1.5 benchmark and the Real5-OmniDocBench Scanning subset, which together support our three dataset configuration experiments.
OmniDocBench v1.5. OmniDocBench [25] is the primary benchmark used throughout this work. Version 1.5 comprises 1,355 bilingual PDF pages spanning nine document types; academic papers, textbooks, financial reports, research reports, slides, books, newspapers, exam papers, and handwritten notes, across five layout types. Pages are annotated at 19 block-level categories and 15 attribute labels, enabling end-to-end, module-level, and attribute-stratified evaluation.
Real5-OmniDocBench (Scanning). Real5-OmniDocBench [39] performs a full-scale, one-to-one physical reconstruction of all 1,355 OmniDocBench v1.5 pages across five real-world degradation conditions: Scanning, Warping, Screen-Photography, Illumination, and Skew. Only the Scanning condition, which represents the most common real-world document capture scenario, is used in the study. The one-to-one correspondence with OmniDocBench v1.5 is critical to our experimental design: because every scanned page shares its ground truth annotation with the digital counterpart, differences in model rankings between the two datasets can therefore be attributed to capture condition alone.
Models
We evaluate 16 models spanning specialist document and general-purpose VLMs Table 1.
Pipeline-based parsers. Docling [14] chains layout analysis, table recognition, and reading order prediction to produce structured markdown.
Specialist document VLMs. This group spans a range of scales and architectures: MinerU2.5 [23] (1.2B), GOT-OCR2 [33] (580M), GLM-OCR [9] (0.9B), PaddleOCR-VL-1.5 [6] (0.9B), Dolphin 1.5 [11] (0.5B), MonkeyOCR-Pro-3B [20] (3B), DeepSeek-OCR 2 [34] (3B), Chandra 2 [7] (5B), HunyuanOCR [13] (1B), Youtu-Parsing [3] (2.5B), dots.ocr [18] (1.7B), and RolmOCR [1] (8B), a fine-tuned Qwen2.5-VL-7B-Instruct variant. All are trained specifically for structured document output and produce markdown.
General-Purpose VLMs. We evaluate three closed-source frontier models via API using the inference scripts and prompts provided in the OmniDocBench repository: GPT-5.5 [24], Gemini 2.5 Pro [5], and Gemini 3.1 Flash [12]. These serve as general-purpose baselines, allowing us to assess whether ranking instability is specific to specialized document parsing models or reflects a property of the evaluation pipeline independent of model class.
A complete summary of model families, parameter scales, open-source availability, and output formats is provided in Table 1.
Table 1: Models evaluated in RankShift. Type: P = Pipeline, S-VLM = Specialist VLM, G-VLM = General-Purpose VLM. Output format indicates the primary structured output produced.
| Model | Type | Params | Open | Doc-Specialised | Output |
| Docling | P | – | ✓ | ✓ | Markdown |
| MinerU2.5 | S-VLM | 1.2B | ✓ | ✓ | Markdown |
| Chandra 2 | S-VLM | 5B | ✓ | ✓ | Markdown |
| DeepSeek-OCR 2 | S-VLM | 3B | ✓ | ✓ | Markdown |
| Dolphin 1.5 | S-VLM | 0.5B | ✓ | ✓ | Markdown |
| dots.ocr | S-VLM | 1.7B | ✓ | ✓ | Markdown |
| GLMOCR | S-VLM | 0.9B | ✓ | ✓ | Markdown |
| GOT-OCR2.0 | S-VLM | 580M | ✓ | ✓ | MD/LaTeX/HTML |
| Hunyuan OCR | S-VLM | 1B | ✓ | ✓ | Markdown |
| MonkeyOCR-pro-3B | S-VLM | 3B | ✓ | ✓ | Markdown |
| PaddleOCR-VL-1.5 | S-VLM | 0.9B | ✓ | ✓ | Markdown |
| RolmOCR | S-VLM | 8B1 | ✓ | ✓ | Markdown |
| Youtu-Parsing | S-VLM | 2.5B | ✓ | ✓ | Markdown |
| Gemini 2.5 Pro | G-VLM | – | × | × | Markdown |
| Gemini 3.1 Flash | G-VLM | – | × | × | Markdown |
| GPT-5.5 | G-VLM | – | × | × | Markdown |
Table 1. Models evaluated in RankShift. Type: P = Pipeline, S-VLM = Specialist VLM, G-VLM = General-Purpose VLM. Output format indicates the primary structured output produced.
Experimental Setup
Experiment 1 — Dataset Dimension. We examine ranking stability under three types of dataset variation. First, page type: we partition OmniDocBench into its nine page-type strata (e.g. academic papers, newspapers, textbooks) and measure pairwise rank stability between every pair comparisons). Second, layout: we partition the same pages into five layout strata (e.g. single-column, double-column, three-column) and repeat the analysis ( comparisons). Third, capture condition: we compare rankings on the original digital pages against the physically re-scanned Real5 pages. Strata are disjoint, so partition-based rankings are derived from entirely different pages, whereas the Real5 comparison holds page content fixed and varies only how the pages were captured. Table A1 in the appendix lists all configurations.
| Variation | Configurations | Comparisons | Evidence relationship |
| Page type | PPT2PDF; academic literature; book; colorful textbook; exam paper; magazine; newspaper; note; research report | All pairwise (36) | Disjoint page subsets |
| Layout | 1+ column; double column; other layout; single column; three column | All pairwise (10) | Disjoint page subsets |
| Capture condition | OmniDocBench v1.5 digital pages; Real5 Scanning reconstructions | Digital vs. scanned (1) | Paired versions of the same 1,355 pages |
Table A1.
Experiment 2 — Alignment Dimension. We compare six alignment strategies, formed by crossing three matching methods with two ground-truth formats. In the four segment-level conditions, predictions are segmented on double line breaks before matching; the methods differ in how those segments are then matched to ground-truth segments: quick_match (adjacency-search-based truncation and merging of adjacent predicted segments, with hybrid formula–text matching to reduce scoring sensitivity to unicode formula output), simple_match (one-to-one assignment without truncation or merging), and no_split (whole-page concatenation into a single string, with no block-level alignment or attribute-level scoring). Each method is applied under two ground-truth formats: end-to-end (E2E), which scores against OmniDocBench’s JSON annotations, and markdown-to-markdown (M2M), which scores against their Markdown rendering. This yields pairwise comparisons.
The quick_match condition reported across all experiments uses the v1.6 implementation, which incorporates MGAM [32]. The redesign applies only to quick_match: its prediction-side granularity search presupposes that predicted segments can be merged or split to fit the ground truth, which simple_match and no_split exclude by definition. The simple_match and no_split procedures are therefore unchanged from v1.5.
Experiment 3 — Metric Dimension. Under a fixed alignment (quick_match E2E), we compare three scoring metrics: normalised edit distance (NED, reported as ), BLEU, and METEOR. All three are computed on the same aligned text segments, so rank differences reflect metric sensitivity rather than matching or pipeline choices.
Metric underlying reported rankings. Although OmniDocBench reports a composite score averaging text edit distance, table TEDS and formula CDM [31], all rankings analysed in the main text are derived from text edit distance alone, unless stated otherwise. This is because tables and formulas are extracted into separate categories before the text matcher runs [25]: the merging and splitting that distinguishes the alignment conditions acts only on the remaining text paragraphs. Reporting a composite would therefore dilute the manipulation.
Alignment Signatures
To characterise where alignment strategies disagree, we represent every complete page×model row by a six-character alignment signature. Page-level NED scores under the three E2E and three M2M alignments are discretised into zero, low, medium, or high buckets and concatenated in fixed order. Across 20,610 complete rows, this produces 301 observed signatures; the 20 most frequent cover 78.5% of rows (Figure B10).

Figure B10. Alignment signatures used in the Experiment 2 diagnostic procedure: the 20 most frequent patterns, covering 78.5% of complete page×model rows. Each row is one signature, with columns in the fixed order E2E quick_match (), simple_match () and no_split (), followed by the M2M counterparts , and .
We combine signature frequency, score-cluster shape, corpus-level text statistics, and stratified manual review to associate recurrent E2E–M2M disagreement patterns with candidate failure mechanisms. Appendix C:2 provides the complete construction, thresholds, auxiliary labels, sensitivity analysis, and review protocol.
Stability Measures
We measure ranking instability between any two conditions and using three complementary statistics computed on the vector of per-model mean scores.
Kendall’s measures global rank correlation ( indicates identical ordering; is consistent with statistical independence).
Top- churn rate quantifies instability at the top of the leaderboard:
where is the set of top- models under condition . A churn rate of 0 indicates the top- set is unchanged; 1 means complete turnover.
Pairwise reversal rate counts what fraction of model pairs have their relative ordering reversed between and :
Bootstrap Confidence Intervals
All three statistics are accompanied by 95% bootstrap confidence intervals obtained from iterations. In each iteration, pages are resampled with replacement and per-model means are recomputed with multinomial weights. For conditions sharing the same page set (cross-dataset, Experiment 3), a single resample is applied to both conditions simultaneously, preserving page-level pairing. For conditions covering disjoint page sets (strata comparisons), the two strata are resampled independently, reflecting the true sampling uncertainty for each. For mean across strata, bootstrap resamples are drawn at the model level.
Results
Ground-Truth Format Substantially Reorders Rankings Under Segment-Level Matching
Ranking stability within a single ground-truth format, across matching strategies. Varying only the matching method within a single ground-truth format produces moderate disagreement. Against the JSON annotations, ranges from 0.550 to 0.767, with reversal rates of 11.7%–20.0%; against the Markdown rendering, agreement is comparable (–0.817), with reversal rates of 9.2%–16.7%. None of these six reversal rates are significant after FDR correction, indicating that they are consistent with ordinary variation within a fixed format (Table 2; Appendix Table A3).
| Dimension | Comparison | Kendall’s [95% CI] | Category |
| Alignment | quick_match vs simple_match (E2E) | 0.767 [0.575, 0.929] | E2E vs E2E |
| Alignment | quick_match vs no_split (E2E) | 0.550 [0.229, 0.821] | E2E vs E2E |
| Alignment | simple_match vs no_split (E2E) | 0.583 [0.256, 0.860] | E2E vs E2E |
| Alignment | quick_match vs no_split (M2M) | 0.817 [0.593, 0.965] | M2M vs M2M |
| Alignment | simple_match vs no_split (M2M) | 0.683 [0.449, 0.892] | M2M vs M2M |
| Alignment | quick_match vs simple_match (M2M) | 0.667 [0.368, 0.911] | M2M vs M2M |
| Alignment | quick_match (E2E) vs quick_match (M2M) | 0.400 [-0.037, 0.772] | E2E vs M2M |
| Alignment | quick_match (E2E) vs simple_match (M2M) | 0.233 [-0.276, 0.699] | E2E vs M2M |
| Alignment | quick_match (E2E) vs no_split (M2M) | 0.383 [-0.071, 0.776] | E2E vs M2M |
| Alignment | simple_match (E2E) vs quick_match (M2M) | 0.367 [-0.009, 0.682] | E2E vs M2M |
| Alignment | simple_match (E2E) vs simple_match (M2M) | 0.300 [-0.207, 0.741] | E2E vs M2M |
| Alignment | simple_match (E2E) vs no_split (M2M) | 0.383 [-0.037, 0.736] | E2E vs M2M |
| Alignment | no_split (E2E) vs quick_match (M2M) | 0.717 [0.474, 0.912] | E2E vs M2M |
| Alignment | no_split (E2E) vs simple_match (M2M) | 0.583 [0.274, 0.852] | E2E vs M2M |
| Alignment | no_split (E2E) vs no_split (M2M) | 0.767 [0.504, 0.948] | E2E vs M2M |
| Dataset | OmniDocBench v1.5 vs Real5 (E2E quick_match NED) | 0.800 [0.589, 0.947] | Cross-dataset |
| Dataset | Mean tau across page-type (data_source) strata, E2E quick_match | 0.475 [0.279, 0.646] | Intra-dataset |
| Dataset | Mean tau across layout strata, E2E quick_match | 0.490 [0.247, 0.686] | Intra-dataset |
| Metric | NED vs BLEU (E2E quick_match) | 0.700 [0.463, 0.874] | — |
| Metric | NED vs METEOR (E2E quick_match) | 0.767 [0.486, 0.945] | — |
| Metric | BLEU vs METEOR (E2E quick_match) | 0.867 [0.720, 0.965] | — |
Table 2. Pairwise Kendall’s between model rankings (mean page-level scores). 95% CIs from bootstrap resampling of models (), replicates.
Ranking stability across ground-truth formats, within a single matching strategy. Three pairs hold the matching method fixed and vary only the ground-truth format, isolating its effect. Under no_split, where each page is scored as a single concatenated string, the two formats agree closely (, reversal rate 10.0%, not significant after FDR correction). Under the two segment-level methods, agreement collapses: for quick_match and for simple_match. Both formats are derived from the same annotations, and the divergence appears only when predictions are matched segment by segment. This is therefore the leaderboard-level consequence of the matching-strategy bias identified by the OmniDocBench authors [23].
Across the four pairs in which both conditions match at segment level, mean Kendall’s is approximately 0.33 and the mean pairwise reversal rate is 31.7%. The most unstable pair, E2E quick_match versus M2M simple_match, reverses 35.8% of model pairs (95% CI [30.0%, 39.2%]) and has a top-5 churn rate of 0.60, meaning that three of the five models in the top set change. Table 2 summarises all pairwise values, and Appendix C.3 reports the corresponding reversal-rate significance tests.
Diagnostic Protocol
The question of which evaluation protocol best agrees with human judgment of parsing quality remains open to investigation. Each appears to have distinct strengths and weaknesses that practitioners, benchmark creators, and OCR developers should treat explicitly. We therefore frame the E2E–M2M score gap as a candidate diagnostic signal, in which the direction and shape of disagreement may indicate which failure mode dominates.
On OmniDocBench v1.5, recurrent E2E-favouring families account for approximately 25% of complete page × model rows, compared with approximately 0.8% for the M2M-favouring families. Our manual review suggests that E2E is comparatively lenient on extra or loosely aligned text, whereas M2M can avoid some region-level equation, layout, or title-matching failures. The asymmetry may therefore reflect corpus composition rather than any general protocol superiority. Appendix C.3 reports the supporting signature clustering, mechanism summaries, manual-review protocol, and diagnostic tables.

Figure 2. Experiment 2 bump chart showing rank trajectories under alignment variation.
Dataset Composition Produces Instability Across Page-Type and Layout Strata
Page-type stratification. Splitting OmniDocBench by document type reveals substantially greater instability than the cross-dataset comparison. Under E2E quick_match NED, the mean agreement across all page-type pairs is , with individual comparisons ranging from for academic literature versus research report to for PPT2PDF versus magazine. Academic literature is a systematic outlier, producing the lowest agreement with nearly every other stratum; its near-zero correlation with research reports indicates that these two text-dense document types induce almost unrelated model orderings. This heterogeneity is obscured by aggregate results: dots.ocr ranks first overall but only tenth within the academic_literature stratum, a nine-position difference.
Layout-type stratification. Layout strata exhibit a similar degree of instability (mean , range [0.267, 0.600] under E2E quick_match NED). The least stable comparison is 1+ column versus three-column (), partly reflecting the small size of the three-column stratum (52 pages). The most stable pairs reach only , including 1+ column versus double column and other_layout versus three-column. Stratifying by layout can again reveal strengths the aggregate conceals: Dolphin 1.5 ranks 11th overall but fourth on three-column pages.
Cross-dataset (OmniDocBench vs. Real5 Scanning). For text-based NED, rankings are broadly preserved when moving from digital PDFs to scanned reconstructions (, 95% CI [0.589, 0.947]; pairwise reversal rate = 0.100, CI [0.067, 0.133]). However, there is still some movement. The top-5 set is not fully preserved (churn rate = 0.20): dots.ocr drops out and DeepSeek-OCR 2 enters. Additionally, five models shift by two or more positions: dots.ocr falls from 1st to 6th under Real5, MonkeyOCR-pro-3B moves from 12th to 14th, while PaddleOCR-VL-1.5, MinerU 2.5 and RolmOCR each improve by two ranks.
Metric Choice Is the Most Stable Dimension
Of the three evaluation dimensions, metric choice produces the least instability. All three pairwise comparisons yield : BLEU versus METEOR achieves the highest agreement (), followed by NED versus METEOR () and NED versus BLEU (), giving a mean of .

Figure 3. Rank-displacement heatmap for Experiment 1 across layout strata. Each cell shows how many leaderboard positions a model gains when ranked on B-pages instead of A-pages (rank A − rank B). (blue) means the model moved 12 places toward #1; (red) means it fell 3 places.
Despite strong overall correlation, individual models still change position under different metrics. dots.ocr leads under NED, whereas Youtu-Parsing leads under both BLEU and METEOR, rising three places from fourth, suggesting its outputs are rewarded more by n-gram overlap than by character-level edit distance. Hunyuan OCR moves from eighth under NED to fifth under BLEU, while MinerU 2.5 moves in the opposite direction, from fifth to seventh. Therefore even the most stable dimension tested produces measurable ranking instability, indicating that leaderboard positions are better read as approximate than as fixed.
Discussion
Leaderboard Position Is Highly Conditional
A model’s rank on a document parsing benchmark reflects not only its capability, but also the evaluation configuration under which it was assessed. The same predictions, scored under different alignment strategies, dataset compositions, or metrics, can produce substantially different orderings. A clear illustration of this is the reversal of rankings between GLMOCR and RolmOCR. Under E2E quick_match, GLMOCR ranks 2nd while RolmOCR ranks 14th. Under M2M simple_match, the positions invert, with GLMOCR falling to rank 13 and RolmOCR rising to rank 5. The direction of this reversal holds across every E2E/M2M pair including the no_split pair, where the two formats on average agree closely (Figure B9), so it reflects the ground-truth format rather than any particular matching method: GLMOCR’s output aligns more closely with OmniDocBench’s JSON annotations, while RolmOCR’s aligns more closely with their Markdown rendering. Applying the diagnostic protocol (Appendix § 9:§ 3) yields a potential explanation: GLMOCR produces substantially more formatting overhead, which region-based matching against the JSON annotations does not heavily penalise, whereas RolmOCR’s segment boundaries potentially diverge from the annotated blocks, which can cost it under JSON matching but not when the page is compared as Markdown. Neither ranking has been shown to be wrong; each reflects a different, equally defensible assumption about what constitutes correct output.

Figure B9. Focused bump chart comparing RolmOCR and GLMOCR across the six alignment settings in Experiment 2. GLMOCR ranks higher under every E2E condition and RolmOCR under every M2M condition, reversing their relative order.
For practitioners, a profile of scores across reasonable design choices is therefore more informative than a single ordering, particularly when the differences can be read against candidate failure modes: it indicates which configurations a model performs well under, and so which document types and failure tolerances it suits.
Segment-Level Matching Amplifies the Effect of the Ground-Truth Format
The prediction segments are identical under both ground-truth formats; what differs is how the reference is divided. The Markdown rendering merges, omits and rewrites some annotated blocks, whereas the JSON format excludes eleven categories from the text score. Segment-to-segment matching has no slack for such differences, since a correctly parsed segment paired with the wrong reference can score near zero, whereas concatenating the page removes the pairing step entirely. This is the leaderboard-level consequence of the matching-strategy bias identified by the OmniDocBench authors [32].
MGAM targets precisely this step, searching for an optimal segmentation granularity on the prediction side while holding the ground truth fixed. Even with MGAM applied, however, the two ground-truth formats agree at only under quick_match, so the choice of reference format remains consequential after the redesign.
Limitations and Future Work
Validation of the diagnostic protocol. The mapping from signature families to failure modes in § 4.2 is derived from score-signature clustering, aggregate text statistics, and targeted manual review of exemplar pages. It is not validated against a held-out labelled sample, so we report the mechanisms behind each family but not the precision with which a signature predicts the dominant failure mode on an unseen page. An annotation study, labelling a sample of pages with their dominant failure mode and measuring agreement with the predicted signature family, would turn the protocol from a descriptive mapping into an instrument with known error rates. Relatedly, our characterisation covers the most frequent families; the long tail of the signature vocabulary (301 signatures, of which the top 20 account for 78% of rows) remains unexamined.
Benchmark coverage. Our study is centred on OmniDocBench v1.5, which to our knowledge is the only document parsing benchmark exposing configurable alignment; benchmarks outside its ecosystem fix a single alignment procedure, or bypass it altogether, so the controlled comparison cannot currently be run elsewhere. Extending RankShift to inherit benchmarks that retain both alignment options (e.g. MDPBench, MPDocBench-Parse) would test whether the relative importance of alignment, dataset, and metric effects holds under different corpus compositions. This bears directly on our diagnostic protocol: the E2E-favouring signatures that dominate here reflect the document types present in OmniDocBench, and a math- or slide-heavy corpus should shift the balance toward the M2M-favouring families we observe only rarely. That the comparison is unavailable elsewhere is itself an argument for exposing alignment as an explicit, configurable variable.
Conclusion
Document parsing benchmarks are used to compare model performance, but rankings are conditional on evaluation design. RankShift shows that these design choices do not contribute equally: alignment strategy and dataset composition substantially reorder model rankings, while metric choice leaves them largely intact. Disagreement across alignment strategies is also potentially informative, since a model’s profile of scores may indicate which failure modes dominate its output. We therefore propose that benchmark releases report the evaluation configuration alongside results, together with rank-stability profiles under reasonable design choices.
Additional Tables
Supporting tables for the results reported in the main text are provided below.
Table A1: Dataset configurations used in Experiment 1. Partition-based comparisons use disjoint page subsets, while the capture-condition comparison pairs two versions of the same pages.
Table A2: Experiment 1 model rankings on OmniDocBench v1.5 vs. Real5 Scanning ( models, E2E quick_match text NED). = OmniDocBench rank − Real5 rank (1 = best); negative means a worse rank under Real5.
| Model | OmniDocBench rank | Real5 rank | Real5 position |
| dots.ocr | 1 | 6 | −5 |
| GLMOCR | 2 | 2 | 0 |
| PaddleOCR-VL-1.5 | 3 | 1 | +2 |
| Youtu-Parsing | 4 | 4 | 0 |
| MinerU2.5 | 5 | 3 | +2 |
| DeepSeek-OCR 2 | 6 | 5 | +1 |
| Chandra 2 | 7 | 7 | 0 |
| Hunyuan OCR | 8 | 9 | −1 |
| Gemini 3.1 Flash | 9 | 8 | +1 |
| Gemini 2.5 Pro | 10 | 11 | −1 |
| Dolphin 1.5 | 11 | 10 | +1 |
| MonkeyOCR-pro-3B | 12 | 14 | −2 |
| GPT-5.5 | 13 | 13 | 0 |
| RolmOCR | 14 | 12 | +2 |
| Docling | 15 | 15 | 0 |
| GOT-OCR2.0 | 16 | 16 | 0 |
Table A2.
Statistical Significance of Rank Reversals
For Experiment 2 we report formal significance testing at three complementary levels, each with Benjamini–Hochberg FDR correction at .
Table A3: Alignment-pair rank reversal rates ( models, 120 model pairs per alignment pair). One-sided binomial tests vs. within-E2E baseline (); -values from Benjamini–Hochberg FDR over 15 alignment pairs (). 95% CIs from page bootstrap. An asterisk indicates significance after FDR correction ().
| Comparison | Rate | 95% CI | -value |
| Cross-paradigm (E2E ↔ M2M) | |||
| E2E quick_match vs. M2M simple_match | 35.8% | [30.0%, 39.2%] | * |
| E2E simple_match vs. M2M simple_match | 35.0% | [27.5%, 36.7%] | * |
| E2E simple_match vs. M2M quick_match | 30.0% | [23.3%, 32.5%] | 0.001* |
| E2E simple_match vs. M2M no_split | 29.2% | [21.7%, 31.7%] | 0.002* |
| E2E quick_match vs. M2M no_split | 28.3% | [21.7%, 31.7%] | 0.003* |
| E2E quick_match vs. M2M quick_match | 25.8% | [20.8%, 29.2%] | 0.018* |
| E2E no_split vs. M2M simple_match | 20.0% | [15.8%, 26.7%] | 0.363 |
| E2E no_split vs. M2M quick_match | 12.5% | [8.3%, 17.5%] | 0.986 |
| E2E no_split vs. M2M no_split | 10.0% | [5.8%, 15.8%] | 0.986 |
| Within E2E | |||
| E2E quick_match vs. E2E simple_match | 11.7% | [7.5%, 14.2%] | 0.986 |
| E2E quick_match vs. E2E no_split | 20.0% | [15.0%, 25.0%] | 0.363 |
| E2E simple_match vs. E2E no_split | 18.3% | [14.2%, 23.3%] | 0.475 |
| Within M2M | |||
| M2M quick_match vs. M2M simple_match | 16.7% | [15.8%, 24.2%] | 0.372 |
| M2M quick_match vs. M2M no_split | 9.2% | [6.7%, 14.2%] | 0.986 |
| M2M simple_match vs. M2M no_split | 15.8% | [13.3%, 20.8%] | 0.459 |
Table A3.
Alignment-pair reversal rate. For each pair of alignments, we count how often two models’ relative order on a page reverses between the alignments (120 model pairs per alignment pair for 16 models) and test whether cross-paradigm reversal rates exceed the mean within-E2E baseline (), which captures normal ranking noise under a fixed protocol. We use one-sided binomial tests with BH-FDR correction over the 15 alignment pairs. Six of the nine cross-paradigm (E2E↔M2M) pairs show significantly elevated reversal rates (25–36% vs. the within-protocol baseline); none of the six within-paradigm pairs do. Rank instability is therefore protocol-specific rather than general noise. All three nonsignificant cross-paradigm comparisons involve E2E no_split; the lowest rate, E2E no_split vs. M2M no_split (10%), is expected because both alignments use whole-page matching, so the paradigm distinction largely collapses.
Per-model rank displacement. Page-bootstrap tests (2,000 resamples) confirm that individual models shift rank systematically across alignments: 78 modelalignment-pair displacements are FDR-significant, including 51 under cross E2E↔M2M pairs. For example, RolmOCR improves by eleven ranks under M2M no_split vs. E2E quick_match, whereas GLMOCR worsens by eight ranks under the same pair.
Model-pair page flips. At page granularity, 869 of 1,800 model-pairalignment-pair combinations show significant head-to-head flip rates under two-sided binomial tests against the pair-mean baseline.
Together, these tests establish that the reversals are systematic and statistically robust rather than artifacts of indistinguishable neighbouring models.
Additional Figures
This appendix presents supplementary figures for the three instability experiments.

Figure B1. Cross-dataset rank displacement from OmniDocBench v1.5 to Real5 Scanning in Experiment 1. Values report OmniDocBench rank minus Real5 rank: positive (blue) values indicate movement toward rank 1 on Real5, negative (red) values indicate a lower rank on Real5, and zero indicates no change.

Figure B2. Pairwise reversal-rate heatmap for Experiment 1. Each cell gives the share of the 120 model pairs whose ordering reverses between two of the nine OmniDocBench page-type strata (36 comparisons).

Figure B3. Full rank heatmap for Experiment 1: each cell represents a model’s rank within each of the nine OmniDocBench page-type strata, where 1 is the best-scoring model on that page type and 16 the worst.

Figure B4. Full rank heatmap for Experiment 1: each cell gives a model’s rank within one of the five OmniDocBench layout strata, where 1 is the best-scoring model on that layout type and 16 the worst.

Figure B5. Top-1 churn across Experiment 1 comparison settings. Each cell indicates whether the highest-scoring model differs between two page-type strata: 1.00 for a different top-ranked model, 0.00 for the same.

Figure B6. Top-3 churn across Experiment 1 comparison settings. Each cell gives the proportion of the top-three set that changes between two page-type strata, from 1.00 where none of the three models are retained to 0.00 where all top-three models are the same.

Figure B7. Top-5 churn across Experiment 1 comparison settings. Each cell gives the proportion of the top-five set that changes between two page-type strata, ranging from 0.20, where one of the five models differs, to 0.80, where four differ.

Figure B8. Cross-dataset rank scatter for Experiment 1, comparing model ranks on OmniDocBench v1.5 with the Real5 Scanning subset.

Figure B11. Experiment 3 bump chart showing each model’s rank trajectory across the three scoring metrics: NED, BLEU and METEOR. Rankings are largely preserved, though the top-ranked model changes, with dots.ocr leading under NED and Youtu-Parsing under both BLEU and METEOR.

Figure B12. Heatmap of pairwise rank agreement across the three Experiment 3 scoring metrics. All three pairs show high agreement, with Kendall’s between 0.700 and 0.867.
Detailed Methodology
This appendix provides additional methodological details and sensitivity analyses supporting the main and supplemental experiments.
Implementation Details
All experiments were run on NVIDIA H200 GPUs. Inference followed the official OmniDocBench evaluation scripts where available; open-source VLMs were served using vLLM [16] where supported.
API models. GPT-5.5, Gemini 2.5 Pro, and Gemini 3.1 Flash were queried using OmniDocBench’s reference inference scripts, with the benchmark’s fixed system prompt instructing the model to output page content in Markdown. Only the output token cap was set; temperature and sampling parameters were left at provider defaults.
Prompt standardisation. All VLMs received the same page image and a standardised prompt requesting structured markdown output. No model-specific prompt engineering was applied beyond each model’s official inference script in OmniDocBench; where no official script existed, the prompt followed the conventions of the closest comparable model.
Alignment Signature Construction
Unit of analysis and scores. For every model and page , we collect the page-level NED score ( normalised edit distance) under each of the six Experiment 2 alignments. We retain only rows with a valid score under all six alignments ( page×model rows for 16 models).
Discretisation. Each score is mapped to one of four buckets:
| Bucket | NED range | Label | Interpretation |
| 0 | zero | Empty or collapsed match | |
| L | low | Poor page-level match | |
| M | medium | Partial match | |
| H | high | Strong match |
Table 1 (PDF p. 27).
Cutoffs were chosen a priori for interpretability rather than from corpus quantiles. Under the pooled 16-model distribution they yield bucket fractions 0: 2.6%, L: 8.0%, M: 27.2%, and H: 62.2%. Alignment-specific distributions differ markedly: E2E quick_match is heavily right-skewed (median NED = 0.993; 86.7% H), whereas M2M simple_match is more dispersed (median = 0.836; 48.0% H and 34.2% M). Fixed thresholds therefore encode a common absolute notion of “good page match” rather than per-alignment relative standing.
We did not rerun the full signature analysis under alternative cutoffs. Qualitatively, the conclusions appear robust to modest boundary shifts, since cross-paradigm gaps are large relative to bucket width on the affected rows, and the top-20 signatures and E2E-high/M2M-low families persist under ±0.05 perturbations of the L/M boundary in spot checks. Perference-based bucketing would increase the observed signature vocabulary without necessarily changing the dominant families.
Signature construction. Buckets are concatenated in the fixed order
The first three positions denote E2E quick_match, simple_match, and no_split; the final three denote the corresponding M2M methods. For example, HHMHHH denotes high scores under every condition except E2E no_split, while HHHMHO denotes a collapsed M2M no_split slot. The vocabulary has at most patterns, of which 301 are observed. Bucketing the NED scores groups rows into recurring types of case, such as those scoring high under every alignment or high under E2E and low under M2M. These groups can be ranked by frequency and inspected as a set, which continuous scores would make impractical since exact repeats are rare.
Derived structure. Each row also receives a score cluster shape, describing how raw scores group when rounded to three decimals (e.g. identical, E2E M2M, E2E split, M2M split, or loner), and a uniformity tier that distinguishes numerically identical from alignment-sensitive rows. At page level, we compute modal-signature fraction and normalised Shannon entropy. We flag mechanism-goldmine pages where at least 65% of models share one signature, entropy is at most 0.55, and a consensus bad slot (0 or L) appears in at least 50% of model rows.
Failure-mode analysis. We proceed in four stages:
Rank exact signatures by frequency and test model enrichment in each top pattern using one-sided binomial tests with Benjamini–Hochberg FDR correction. The top 20 signatures cover 78.5% of rows (Figure B10).
Assign rule-based tags (e.g.,
all_highandmn_zero) and group cross-paradigm families by the E2E–M2M gap direction.
Compare family-level prediction length, format-character ratio, header density, and M2M insertion dominance with corpus baselines and test model-level correlations.
Review up to five distinct pages per (top-20 signature, cluster-shape) cell ( 490 exemplars), plus models diverging from the modal signature on approximately 19 mechanism-goldmine pages. Reviewers check for excess or hallucinated content, formatting overhead, region-collapse artefacts, and layout mis-pairing.
Diagnostic Protocol Details
The diagnostic reading combines automatic alignment-signature clustering, cross-alignment mechanism summaries, and targeted manual review. Tables C1 and C2 summarise frequent recurring families on OmniDocBench v1.5. Frequencies are the share of listed exemplar signatures in each row; the rows are illustrative families rather than an exhaustive partition.
E2E-favouring signatures listed in Table C1 account for approximately 25% of pagemodel rows (12% excess-content and 13% format-overhead families), compared with approximately 0.8% for the listed M2M-favouring signatures. The broader pattern “E2E all-high, M2M not all-high” covers approximately 17% of rows, whereas M2M-favouring rows with all-E2E 0/L slots account for approximately 0.2%.
| Observed pattern | Likely cause | What to check | Example signatures | Freq. | Recommendation |
| E2E high, M2M lower | Excess content (hallucinations, captions, duplication) | Pred. length vs. GT; extra spans | HHMMM, HHHMMM, HHMMML, HHHHMM, HHHMML, HHLLL, HHMML | 12% | Report both protocols; inspect extra output when E2E M2M |
| E2E high, M2M low/medium, heavy syntax | Formatting overhead (Markdown/HTML penalised byte-wise) | Headers/HTML; stripped vs. full Markdown | HHHHHM, HHHMHM, HHMMLM, HHHHLH, HLHLHH, HHHMLM | 13% | Pair region-based scores with M2M; report syntax-stripped variants |
Table C1. Diagnostic families where E2E M2M (OmniDocBench v1.5, 16 models; pagemodel rows). Frequencies are the combined share of listed signatures.
This frequency gap does not establish that E2E is globally superior. OmniDocBench is dominated by cases where region-based matching is lenient on extra or loosely aligned text and full-page Markdown comparison is stricter. Math-heavy benchmarks may produce more M2ME2E equation-mispairing cases such as LLLHHH or LLMHMM; title-heavy slides or worksheets may produce more OO*HHH-type region-collapse cases.
| Observed pattern | Likely cause | What to check | Example signatures | Freq. | Recommendation |
| E2E low, M2M high (math) | Equation/block mis-pair under E2E | Equation blocks in GT | LLLHHH, LLLHMM, LLMHMM | Stratify math-heavy pages; audit block matching | |
| E2E low, M2M high (layout) | Column/reading-order break under concatenation | Block order vs. M2M concatenation | MMMHHH, MMMHMH, MMLHMH, HLMHLH, LLLMML, 00LLLL | Report M2M on layout-heavy pages | |
| E2E title/box slot empty, M2M body correct | Region-slot failure (title dropped) | GT title region vs. E2E match | 00MHHH, 00HHHH, 00MHHM | Inspect region-level matching, not page OCR quality |
Table C2. Diagnostic families where M2M > E2E (same corpus; rare on OmniDocBench v1.5).
References
- [1]Reducto AI. Rolmocr: A faster, lighter open source ocr model. Hugging Face model card, 2025.
- [2]Norah A. Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Z. Alsubaie, Yousef Almushayqih, Faisal Abdulrahman Mirza, Nouf M. Alotaibi, Nora Al-Twaire sh, Areeb Alowisheq, M. Saiful Bari, and Haidar Khan. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781, 2024. doi: 10.48550/arXiv.2402.01781.DOI
- [3]Haoyu Cao et al. Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding. arXiv preprint arXiv:2601.20430, 2026. URL https://arxiv.org/abs/2601.20430.
- [4]Center for AI Safety, Scale AI, and HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature, 649:1139–1146, 2026. doi: 10.1038/s41586-025-09962-4.
- [5]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. doi: 10.48550/arXiv.2507.06261.arxiv.org/abs/2507.06261
- [6]Cheng Cui et al. PaddleOCR-VL-1.5: Towards a multi-task 0.9b VLM for robust in-the-wild document parsing. arXiv preprint arXiv:2601.21957, 2026.arxiv.org/abs/2601.21957
- [7]Datalab. Chandra ocr 2. Hugging Face model card, 2025.
- [8]Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107.07002, 2021. doi: 10.48550/arXiv.2107.07002.DOI
- [9]Shuaiqi Duan et al. GLM-OCR technical report. arXiv preprint arXiv:2603.10910, 2026.arxiv.org/abs/2603.10910
- [10]Maria Eriksson, Erasmo Purificato, Arman Noroozian, João Vinagre, Guillaume Chaslot, Emilia G’omez, and David Fernandez-Llorca. Can we trust AI benchmarks? an interdisciplinary review of current issues in AI evaluation. arXiv preprint arXiv:2502.06559, 2025.
- [11]Hao Feng, Shu Wei, Xiang Fei, Wei Shi, Yingdong Han, Lei Liao, Jinghui Lu, Binghong Wu, Qi Liu, Chunhui Lin, Jingqun Tang, Hao Liu, and Can Huang. Dolphin: Document image parsing via heterogeneous anchor prompting. arXiv preprint arXiv:2505.14059, 2025.arxiv.org/abs/2505.14059
- [12]Google. Gemini 3.1 flash live preview. Google AI for Developers model documentation, 2026. URL https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview. Accessed 2026-08-29.
- [13]Hunyuan Vision Team et al. HunyuanOCR Technical Report. arXiv preprint arXiv:2511.19575, 2025. URL https://arxiv.org/abs/2511.19575.
- [14]IBM Research and Contributors. Docling: Technical report. GitHub repository and technical report, 2024.
- [15]Ihor Kendiukhov. Quantifying ranking instability across evaluation protocol axes in gene regulatory network benchmarking. arXiv preprint arXiv:2603.03493, 2026. doi: 10.48550/arXiv.2603.03493.DOI
- [16]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. arXiv preprint arXiv:2309.06180, 2023. doi: 10.48550/arXiv.2309.06180.arxiv.org/abs/2309.06180
- [17]Yanhang Li, Zhichao Fan, and Zexin Zhuang. SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks. arXiv preprint arXiv:2605.25492, 2026. doi: 10.48550/arXiv.2605.25492.DOI
- [18]Yumeng Li, Guang Yang, Hao Liu, Bowen Wang, and Colin Zhang. dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model. arXiv preprint arXiv:2512.02498, 2025. URL https://arxiv.org/abs/2512.02498.
- [19]Zhang Li, Zhibo Lin, Qiang Liu, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiajun Song, Jiarui Zhang, Xiang Bai, and Yu liang Liu. MDPBench: A benchmark for multilingual document parsing in real-world scenarios. arXiv preprint arXiv:2603.28130, 2026.arxiv.org/abs/2603.28130
- [20]Zhang Li et al. Monkeyocr: Document parsing with a structure-recognition-relation triplet paradigm. arXiv preprint arXiv:2506.05218, 2025.arxiv.org/abs/2506.05218
- [21]Benjamin Marie, Atsushi Fujita, and Raphael Rubino. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. arXiv preprint arXiv:2106.15195, 2021. doi: 10.48550/arXiv.2106.15195.arxiv.org/abs/2106.15195
- [22]Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, 2020.arxiv.org/abs/2006.06264
- [23]Junbo Niu et al. MinerU2.5: A decoupled vision-language model for efficient high-resolution document parsing. arXiv preprint arXiv:2509.22186, 2025. URL https://arxiv.org/abs/2509.22186.
- [24]OpenAI. GPT-5.5 model documentation. OpenAI developer documentation, 2026.
- [25]Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He. OmniDocBench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2412.07626
- [26]Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, 2018.arxiv.org/abs/1804.08771
- [27]Jake Poznanski, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. olmocr: Unlocking trillions of tokens in pdfs with vision language models. arXiv preprint arXiv:2502.18443, 2025.arxiv.org/abs/2502.18443
- [28]Qi Qian, Chengsong Huang, Jingwen Xu, Changze Lv, Muling Wu, Wenhao Liu, Xiaohua Wang, Zhenghua Wang, Zisu Huang, Muzhao Tian, Jianhan Xu, Kun Hu, He-Da Wang, Yao Hu, Xuanjing Huang, and Xiaoqing Zheng. Benchmark² : Systematic evaluation of llm benchmarks. arXiv preprint arXiv:2601.03986, 2026.arxiv.org/abs/2601.03986
- [29]Lin Sun, Dexian Wang, Jingang Huang, Linglin Zhang, Change Jia, Zhengwei Cheng, and Xiangzheng Zhang. When good OCR is not enough: Benchmarking OCR robustness for retrieval-augmented generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 884–894. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.acl-industry.60.DOI
- [30]Rohan Taori, Achal Dave, Vaishaal Shankar, et al. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2007.00644
- [31]Bin Wang, Fan Wu, Linke Ouyang, Zhuangcheng Gu, Rui Zhang, Renqiu Xia, Botian Shi, Bo Zhang, and Conghui He. Image over text: Transforming formula recognition evaluation with character detection matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19681–19690, June 2025. doi: 10.1109/CVPR52734.2025.01833. URL https://ieeexplore.ieee.org/document/11094880.DOI
- [32]Bin Wang et al. MinerU2.5-Pro: Pushing the limits of data-centric document parsing at scale. arXiv preprint arXiv:2604.04771, 2026. doi: 10.48550/arXiv.2604.04771.DOI
- [33]Haoran Wei et al. General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model. arXiv preprint arXiv:2409.01704, 2024.arxiv.org/abs/2409.01704
- [34]Haoran Wei et al. DeepSeek-OCR 2: Visual causal flow. arXiv preprint arXiv:2601.20552, 2026.arxiv.org/abs/2601.20552
- [35]Minglai Yang, Xinyan Velocity Yu, Pengyuan Li, Xinyu Guo, Zhenting Qi, Konwoo Kim, Longtian Ye, Xiaolong Luo, Jinhbe Li, Henry Zhang, Haris Riaz, Xuan Zhang, Yunze Xiao, Bangya Liu, Tom Tang, Yunfei Zhao, Qunshu Lin, Zihan Wang, Minghao Liu, Michael Lingzhi Li, Yilun Du, Jesse Thomason, Rogerio Feris, Alex Pentland, and Zexue He. Dr. DocBench: A comprehensive benchmark for expert-level and difficult document parsing. arXiv preprint arXiv:2606.01393, 2026.arxiv.org/abs/2606.01393
- [36]Boyang Zhang, Sebastián G. Acosta, Preston Carlson, Sacha Bron, Pierre-Loïc Doulcet, Daniel B. Ospina, and Simon Suo. ParseBench: A document parsing benchmark for ai agents. arXiv preprint arXiv:2604.08538, 2026. doi: 10.48550/arXiv.2604.08538.DOI
- [37]Qintong Zhang et al. DOCR-Inspector: Fine-grained and automated evaluation of document parsing with VLM. arXiv preprint arXiv:2512.10619, 2025.arxiv.org/abs/2512.10619
- [38]Bangbang Zhou, Hangdi Xing, Yifan Chen, Jianjun Xu, Qi Zheng, Feiyu Gao, Zhibo Yang, Shuai Bai, Ming Yan, Jieping Ye, and Hongtao Xie. MPDocBench-Parse: Benchmarking practical multi-page document parsing. arXiv preprint arXiv:2605.22100, 2026.arxiv.org/abs/2605.22100
- [39]Changda Zhou et al. Real5-OmniDocBench: A full-scale physical reconstruction benchmark for robust document parsing in the wild. arXiv preprint arXiv:2603.04205, 2026.arxiv.org/abs/2603.04205