betaFeedback

WorksComputer Science

Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling

Abstract

Mathematical modeling turns an open-ended real-world problem into assumptions, a model, a solution, and a defended recommendation, and its answers matter only if they are both close to a credible reference and physically feasible; existing benchmarks either score solver fragments or grade reports with an LLM judge, leaving those numbers unchecked. We introduce TB-MathModel (the Terminal-style Mathematical Modeling Bench), the first benchmark to make complete contest-style modeling problems agentic, verifiable, and terminal at once, over 18 real CUMCM and MCM problems (2023–2025). Each task ships the verbatim statement, official attachments, and figures in a containerized terminal; the agent returns one JSON artifact, scored on the numeric core of the final question against a reproduction of the Outstanding-award paper. Two design choices distinguish it: direction-aware, one-sided scoring, which never penalises beating the anchor, and two-layer feasibility verification—a programmatic gate over six decidable invariant forms plus 624 Lean~4 obligations—with a hard gate that zeroes infeasible answers. Each task is scored against a panel of human outstanding papers rather than one, so a model must be close to many accepted answers; across seven multimodal entries (three seeds each) the strongest reach a multi-reference O-Eval of 64.58%, while the $2.54 open model holds 59.81% at 1/1001/100 the cost. TB-MathModel turns the numeric core of modeling into a reproducible, machine-checkable signal for measuring long-horizon modeling competence and for collecting agent trajectories. The source code is available at https://github.com/wxj630/terminal-bench-math-modeling.

Introduction

Mathematical modeling contests such as CUMCM in China and MCM/ICM internationally ask teams to convert a deliberately under-specified real-world problem into a defensible quantitative answer within days. The deliverable is not a single number but a chain: assumptions →\to model →\to solution →\to sensitivity →\to recommendation. Two properties make it an attractive target for agent evaluation: it is quantitative, because the final question ends in numbers that can be checked, and it is multimodal, because statements reference figures and attachments are spreadsheets and CSVs. It also differs from solving a well-posed problem in where the hard work begins—the model itself must be built, and the same situation admits time-series, optimization, dynamical, graph, or multi-criteria formulations, any of which can be defensible if its assumptions are explicit and its computation credible {(citep)}(Giordano et al., 2014; Meerschaert, 2013). This is what makes modeling a demanding test of long-horizon agents: hold the statement and attachments in context, choose a formulation, write and run code, handle files and units, and report a self-consistent answer.

Existing benchmarks cover adjacent but distinct ground, and none closes the loop we care about. Code, terminal, and assistant benchmarks score repository edits, command-line competence, and tool use {(citep)}(Jimenez et al., 2024; Merrill et al., 2026; Mialon et al., 2024; Yao et al., 2025); closed-form mathematics benchmarks score deduction rather than modeling {(citep)}(Hendrycks et al., 2021; Cobbe et al., 2021; Glazer et al., 2024); operations-research datasets parse word problems into solver-ready formulations {(citep)}(Ramamonjison et al., 2023; AhmadiTeshnizi et al., 2024; Yang et al., 2025). The closest prior work on modeling itself falls into two camps, neither simultaneously agentic, verifiable, terminal, and multimodal. One stays solver-facing and narrow, scoring a single formal artifact {(citep)}(Huang et al., 2025b). The other grades the final report with an LLM judge {(citep)}(Qian et al., 2025; Liu et al., 2025), rewarding fluent prose without establishing that the reported numbers are correct or even admissible. Table 1 makes the gap concrete; TB-MathModel is the first to combine all of these properties.

Benchmark#Tasks#Fig.Evaluation signalAgenticPhysicalLeanMultimodal
Mamo ([12])1209–Solver check✗✗✗✗
ModelingBench ((Qian et al., 2025))68–LLM judge✗✗✗✗
MM-Bench ([17])111–LLM judge✗✗✗✗
TB-MathModel (ours)1821Executable verifier✓✓✓✓

Table 1. The closest mathematical-modeling benchmarks. Only TB-MathModel combines a complete contest problem, an agentic terminal run, an executable verifier, physical-feasibility checking, and a machine-checkable proof, on a multimodal task (“Judge”: LLM grades a report; “Solver”: solver checks one formal artifact).

Closing the gap raises three problems that shape the design. Different teams make different assumptions, so there is no single ground truth: rather than promote one paper to truth, TB-MathModel asks every human expert, scoring each task against a panel of outstanding solutions so that a model must be close to many accepted answers, not match one. Anchoring creates asymmetric failure—a model that does better than the paper can score worse than one that copies it—so TB-MathModel makes each metric’s direction explicit and scores one-sidedly. And a wrong-but-plausible answer is easy to produce (numbers that look reasonable yet violate the problem’s physics), which anchor comparison cannot catch, so TB-MathModel adds an independent feasibility layer.

We present TB-MathModel (Terminal-style Mathematical Modeling Bench), to our knowledge the first benchmark to turn complete contest-style mathematical modeling problems into an agentic, verifiable, terminal evaluation. It contributes:

  1. The first benchmark of its kind, and the 18-task dataset behind it. No prior benchmark turns complete contest-style modeling problems into a containerized terminal task with an executable verifier. TB-MathModel covers CUMCM and MCM 2023–2025 (problems A/B/C each year), with verbatim statements, official attachments, 21 figures cropped from the PDFs, and one JSON artifact per task.

  1. Direction-aware, one-sided scoring with auditable edge-case rules. Per-metric semantic directions (higher/lower_is_better, target_value, exact_value); two reported metrics (O-EVAL, ROBUST BO-EVAL); explicit rules for degenerate bands and unpinned reference units; and contract tests that lock the two defining properties down.

  1. A two-layer feasibility verification design. A programmatic gate covering 18/18 tasks through six decidable invariant forms, and a Lean 4 certificate layer of 624 kernel-checked obligations; the two layers decide every feasibility question a structured artifact can settle, and state exactly where an independent replay takes over.

  1. An empirical study of seven multimodal model/harness entries, each averaged over three seeds, with per-task behaviour, a cost–robustness analysis, and three case studies; we also document the scoring pitfalls we found and fixed and quantify which move the leaderboard.

Against the human panel, the two newest flagship entries, each averaged over three seeds, with per-task behaviour, a cost–robustness analysis, and three case studies; we also document the scoring pitfalls we found and fixed and quantify which move the leaderboard.

Against the human panel, the two newest flagships lead (GPT-6 Astra + codex at O-EVAL 64.58 ± 2.93 and MULTI-BO 1.132, Fable 5.1 + claude-code at 63.11 ± 1.33 and 1.081, means over three seeds), while the $2.54 DeepSeek V4.1 Flash + opencode holds 59.81 ± 1.25 at 1/100 the cost; varying only the harness moves O-EVAL by at most 0.07 pp, so the board measures the model rather than the harness. Per-seed deviations reach 2.9 pp, so adjacent rows are not significantly separated. One case study inverts the usual story: on dragon-dance the models were right and our anchor was wrong—the sharpest argument for independently checkable feasibility.

Related Work

Agent and assistant benchmarks. Software-engineering and terminal agents are measured by repository-level tasks (SWE-bench [14] ) and end-to-end command-line tasks (Terminal-Bench [19]); assistants by tool-use suites such as GAIA [20] and τ\tau-bench (Yao et al., 2025). Machine-learning and data-science agents are measured on Kaggle-style pipelines [3, 9, 8, 1, 15] (Starace et al. 2025; Zhang et al. 2024). These capture tool use and long-horizon execution in a container—the harness we build on—but score a leaderboard position or a rubric over a deliverable, never a modeling artifact, and attach no feasibility contract.

Mathematical and optimization reasoning. Closed-form mathematics benchmarks measure deduction, not modeling [10, 4, 7] (Zheng et al. 2022). Operations-research systems parse word problems into solver-ready formulations and evaluate optimality or feasibility of a given formulation [2, 11, 16] (Ramamonjison et al. 2023; Xiao et al. 2024; Yang et al. 2025; Mostajabdaveh et al. 2025; Xiao et al. 2025). They supply a strong execution-validity precedent—a solver certifies a solution is feasible or optimal *for a stated model—but cannot certify that the model captures the right abstraction, and none requires assumptions, sensitivity analysis, or a defended recommendation.

Why modeling is hard to evaluate, and what TB-MathModel changes

Figure 1. Why modeling is hard to evaluate, and what TB-MathModel changes. (a) In the status quo an LLM judge grades a report on organisation while the numbers are never checked. (b) TB-MathModel runs the problem, attachments, and figures in a terminal container, requires one structured artifact, scores it one-sidedly against an O-award reproduction, and verifies feasibility with a gate and Lean certificates—so a plausible-but-infeasible answer is caught.

Mathematical-modeling benchmarks and agents

The closest work forms a continuum, and none of it is simultaneously agentic, verifiable, and terminal. Mamo ([12]) is solver-grounded: models emit solver-consumable artifacts and the solver is a falsifiable checker, giving a precise but narrow signal on formal fragments rather than whole contest problems. ModelingAgent and its ModelingBench ((Qian et al., 2025)) are report-level: open-ended problems graded by an expert-style LLM judge—ecologically rich but judge-calibration dependent and non-reproducible. MM-Agent and MM-Bench ([17]) add a hierarchical modeling library, yet their primary evaluation remains a report rubric. TB-MathModel keeps the full contest statement and the agentic workflow, pins evaluation to a structured artifact scored by an executable verifier, and adds a feasibility layer neither camp provides.

Verification in evaluation

Theorem provers such as Lean 4 ([5]) with Mathlib ((The mathlib Community, 2020)) have certified competition and research mathematics, and RL over formal proof has produced strong provers ([13] (Xin et al. 2024)). Using a prover not to find answers but to audit a benchmark’s own feasibility decisions is, to our knowledge, new: our Lean layer re-proves the programmatic gate’s invariants, so an assertion becomes a machine-checkable fact ((Sutton, 2001)), at integer-and-rational arithmetic cost the kernel decides without the full library.

Multimodal agents and reproduction-anchored scoring

Tool-using agents increasingly externalize their state for reuse ((Yao et al., 2023; Wei et al., 2022; Schick et al., 2023; Zhou et al., 2026)). TBMathModel makes visual evidence load-bearing—figures are cropped from the official PDFs—so harnesses with and without an image channel can be compared on identical tasks. Anchoring a numerical benchmark on a published solution is standard practice; we show it creates three under-reported hazards (direction asymmetry, degenerate scoring bands, unpinned units) and give auditable rules and tests for each.

The TB-MathModel Benchmark

TB-MathModel (the Terminal-style Mathematical Modeling Bench) turns each contest problem into a self-contained terminal task. Figure 2 summarises the construction pipeline. A build script assembles the official contest material into four parts: (i) an instruction file with the verbatim statement and its figures; (ii) an agent container holding the mirrored attachments and an output schema; (iii) a separate verifier container holding the scoring panel and an executable feasibility check; and (iv) an oracle that reproduces the Outstanding-award (O-award) paper. The agent is dropped into the terminal with this material and must return a single structured artifact. Everything downstream of the artifact—scoring and feasibility—is machine-executable.

Figure 2. The TB-MathModel construction and evaluation pipeline. Build: official statements, attachments, and PDF figures are compiled into a task directory. Run: the agent works inside a network-isolated terminal container and writes one JSON artifact. Evaluate: the artifact is scored by a direction-aware, one-sided panel against an O-award reproduction, and independently checked by a programmatic feasibility gate plus a Lean 4 certificate layer; a failed gate zeroes the task.

Task selection and sources. TB-MathModel comprises 18 tasks: the A, B, and C problems of CUMCM and MCM for each of 2023–2025. We restrict to the last three years on purpose: older problems predate capable LLMs and were calibrated to software-assistant graders, so their task mix and review standard differ from the post-LLM regime, and recent editions are harder and more discriminating. The two contests complement each other—CUMCM is engineering/optimization-heavy and Chinese, MCM is modeling-flavored and English—and taking all three problems of each year covers the standard “A = physical/engineering, B = data/optimization, C = data-science” split without cherry-picking. Statements and attachments are mirrored verbatim; a statement’s figures are cropped from the official PDF (artwork plus its caption) and delivered inline and at stable container paths. 21 unique figures across 6 tasks are load-bearing: the statements reference them for geometry not restated in text (e.g., the dragon-dance U-turn curve, which fixes the final head speed), so a harness without an image channel cannot recover it—hence the multimodal evaluation. TB-MathModel is a pure evaluation set, with no training split or difficulty tiering (Appendix § 8).

The artifact contract and oracle anchors. The scored object is one JSON file per task, whose schema (field names and types only) is shipped to the agent; extra CSV/XLSX/PNG artifacts are not scored. A schema-constrained JSON is what makes an automated score possible, at the cost of mapping a free-form effort onto named numeric fields—a tension we revisit in § 4.3. Each task is anchored to one O-award paper, re-implemented as a script that reads the attachments and emits the same JSON; the reproduction is the normalisation endpoint (O-EVAL = 100%) and sets the per-metric panel, coverage 18/18. Provenance is a first-class field—paper id, PDF/OCR source, and reproduction scope—so any anchor can be re-derived (Appendix § 11); this is what let us catch and correct an infeasible dragon-dance anchor before it reached the board.

Scoring: direction-aware, one-sided, auditable

A naive distance-to-anchor metric silently rewards copying and punishes improvement, so TB-MathModel scores each task through a per-metric panel with an explicit semantic direction. Each task ships a panel of 2–6 scored metrics (60 in total), each with a path into the artifact, an anchor value OO, a direction, and a weight. A metric is higher_is_better, lower_is_better, target_value (closest to OO wins), or exact_value, set from the modeling semantics of the quantity rather than the sign of its value. Directional metrics use an asymmetric, one-sided reward: meeting or beating the anchor never scores below 1.0, so improving on the paper is never punished (full formulas in Appendix § 14).

Three failure modes recur often enough to deserve named rules. (i) Degenerate bands: a band narrower than 2% of the value magnitude cannot discriminate, so we widen it to 5%—this changed one cell (stair-wear, 72.67 → 73.52). (ii) Non-comparable units: when an anchor is an arbitrary raw index with no schema-pinned unit (a host-fish index of 1080), a model reporting the same quantity normalised to [0, 1] is not wrong, so such metrics are dropped for every model and the oracle by a configuration-only rule. (iii) Cardinality mismatch: surplus items are excluded rather than counted missing. Out-of-domain values (a correlation outside [−1, 1], a negative count) score zero, with the reason recorded.

Let RR, FF, OO be the weighted panel raw of a submission, of the DeepSeek V4 Flash baseline, and of the oracle. We report O-EVAL == clamp⁡(R/O,0,1)\operatorname{clamp}(R/O, 0, 1) averaged over tasks (the headline absolute score) and a gap-compensated ROBUST BO-EVAL == clip⁡((R−F)/(O−F),±1)\operatorname{clip}((R-F)/(O-F), \pm1) when the oracle–baseline gap exceeds 0.10, and the clipped raw gain R−FR-F otherwise, with ROBUST BO-EVAL == −1-1 for a missing artifact (full formulas in Appendix § 14). Two properties are locked by contract tests: oracle self-consistency (every reproduction scores exactly 1.0) and one-sidedness (at or above the anchor is never a failure). A failed feasibility gate sets a task’s O-EVAL to 0 and ROBUST BO-EVAL to −1-1.

Multi-references evaluation

Modeling has no single correct answer, so a benchmark that normalises against one paper silently converts one team’s choices into ground truth. TB-MathModel instead asks every human expert. For each task we extracted the final-question values reported by all available Outstanding-award papers for that problem, giving 99 panel papers, 94 of which yield the scored quantity. Coverage is 18/18 tasks and all available outstanding papers are used—1 to 18 per task (median 4), since the number of O-award reports differs by problem. Each paper is one vote.

Let M(t)M(t) be the scored metrics of task tt, P(t)P(t) its paper panel, and Sp⊆M(t)S_p \subseteq M(t) the fields paper pp actually reports. Writing R(m,S)R(m,S) for the weighted panel score of submission mm restricted to SS (weights renormalised over SS), the task score is the panel mean

O-EVAL⁡(m,t)=1∣P≠∅(t)∣∑p∈P≠∅(t)R(m,Sp),(1)\operatorname{O\text{-}EVAL}(m,t)=\frac{1}{\lvert P_{\ne\varnothing}(t)\rvert}\sum_{p\in P_{\ne\varnothing}(t)}R(m,S_p), \tag*{(1)}

where P≠∅(t)={p∈P(t):Sp≠∅}P_{\ne\varnothing}(t)=\{p\in P(t):S_p\ne\varnothing\}. Each paper thus contributes exactly “how close is the model to *this human answer, on the fields that human reported”. Because R(p,Sp)=1R(p,S_p)=1, every paper agrees with itself and the reproduction reproduces the panel. Equal weighting keeps one human solution to one vote, and a paper that reports only a single field acts as a single-field check rather than being discarded—the honest treatment of an incomplete report.

A paper is never silently dropped. When it cannot supply the scored quantity it carries a machine-readable exclusion_reason, of which five forms occur: number_only_in_figure, figure_only, figure_only_logistic_prior, no_numeric_estimate, and different_unit_of_analysis; these account for the 5 of 99 papers outside the panel. Values are never interpolated or adjusted toward the existing anchor, and a paper’s different-but-equivalent unit is converted explicitly and noted. Where a panel is empty—possible when an MCM answer is prose or figure only—the task falls back to the single reproduced anchor, and a missing artifact scores 0, so the denominator is always the fixed 18 tasks.

A panel also makes disagreement between humans visible instead of hiding it behind one anchor. On mcm-2024-c-tennis-momentum, 10 of 11 outstanding papers conclude that momentum is supported while the reproduced anchor—itself an O-award report—concludes it is not; on that task a model scores 0.909 if it answers with the majority and 0.091 if it matches the anchor, so the score reflects which valid statistical test the model chose rather than modeling quality. We mark such literature-split tasks explicitly and exclude them from any headline ranking, while keeping them in the panel: they are the clearest evidence that a single-anchor score would have been arbitrary.

Relative to the single-anchor board, this changes Robust Bo-Eval into a multi-reference gain index against the DeepSeek V4.1 Flash baseline, anchored at 1.0:

MULTI-BO=1+clip⁡(Ot−ft1−ft,−1,+1),(2)\mathrm{MULTI\text{-}BO} = 1 + \operatorname{clip}\left(\frac{O_t-f_t}{1-f_t},-1,+1\right), \tag*{(2)}

with ftf_t the Flash score on task tt and OtO_t the model’s, averaged over tasks; when the remaining headroom 1−ft1-f_t is below 0.10 we fall back to 1+clip⁡(Ot−ft,±0.10)1+\operatorname{clip}(O_t-f_t,\pm0.10), and a missing artifact scores −1-1. A value of 1.0 therefore means “level with Flash”, 1.08 means “closed 8% of the remaining gap to the human panel”; it is a ratio, not a percentage, and is not comparable to O-EVAL.

Feasibility verification: two layers

Proximity to an anchor does not imply that an answer is physically admissible: a submission can match a reference value on one field while violating a stated constraint on another. TB-MathModel verifies feasibility independently of scoring, in two layers with different trust bases.

Layer 1: a programmatic gate over six decidable forms. Every task registers one feasibility check, and the registry covers 18/18 tasks; every hard-error branch falls into one of six decidable forms (Appendix Table 12). Checks are either structural/domain bounds (rates, correlations, probabilities, non-negative counts, distributions summing to 100, peak ≥\ge average) or constraint replay (heliostat rated power ≥\ge 60 MW and installation height ∈[2,6]\in[2,6] m, dragon-dance head speed ×\times amplification ratio ≤\le 2 m/s, a demand-capped revenue ceiling, a union duration ≤\le the sum of its parts). Three contracts keep the gate honest: coverage (a task cannot ship ungated), oracle self-consistency (the O-award reference must pass its own check), and discrimination (synthetic and real-field violations must be rejected, while an empty artifact warns rather than hard-fails). Four constraints that need an independent replay rather than a field check are recorded as whole-task penalties (Appendix § 14).

Layer 2: Lean certificates. To make the gate’s own correctness checkable, Layer 2 re-proves the same invariants as Lean 4 theorems, so the audit rests on the kernel. It generates 624 obligations across all 18 tasks, discharged by decide or omega. No Mathlib is used—the propositions are decidable on exact integers—and values are scaled to integers, so there is no floating-point tolerance; the one exception, a composite index, is declared in the predicate because the tolerance is itself part of the audited assumption. A trust audit (#print axioms) reports only [propext, Quot.sound] or none, with no sorryAx and no native_decide (grep-checked). A discrimination suite mutates one numeric leaf at a time and requires the build to fail and to cite the field; ten mutations across all six forms are rejected. Two soft warnings become proofs: a tourism acceptance index is refuted for the unit convention, and a sustainability score fits neither convention within a declared budget.

Verification scope. The two layers decide every feasibility question that a structured artifact can settle without a model of the physics (Appendix Table 13): unit/value ranges (A) and internal consistency (B). Whether the reported value reflects real physical truth (C)—does the field really produce 60 MW?—is a property of the world rather than of the artifact, so TB-MathModel verifies it where the problem admits an independent replay. Within this scope the layers are exact: any obligation whose evidence is absent is reported as a witnessed gap rather than silently satisfied, so the audit never overstates what it has proved.

Experiments

Setup

We evaluate seven multimodal model/harness entries, whose harness can open the cropped figures. Each entry is run with three seeds per task and reported as a mean with standard deviation; where a model was run at two reasoning tiers we keep the higher-scoring variant. Every row is scored with the same scorer and current anchors, so the board shares one scoring definition (Appendix § 10).

The prompt is the task’s instruction file, passed verbatim: the contest statement, the cited figures inline, and a fixed “Required Benchmark Artifact” block naming the output path (Appendix § 12). Tasks run in Docker containers—python:3.12-slim with a fixed scientific stack, internet disabled, model traffic through a local retry proxy—so an agent cannot fetch solutions or install packages at run time; limits are 2 CPUs, 8 GB, no GPU, and a 4-hour timeout (Appendix § 9). Each multimodal entry is run three times per task and reported as a mean with standard deviation; the one well-posed task, dragon-dance, is additionally probed four times, where the answers agreed to machine precision (std=4.6×10−164.6 \times10^{-16}).

Main result: the multimodal board

Table 2 is the main result: the seven multimodal entries on all 18 tasks, each a three-seed mean (± std); the per-entry configuration is in Appendix Table 7. O-EVAL is the primary metric: the multi-reference score of § 3.2, i.e. the mean over the human paper panel of how close the model is to each accepted answer. MULTI-BO is the secondary, relative index of that same score against the DeepSeek V4.1 Flash baseline, fixed at 1.000 cell-by-cell: 1.08 means the model closed 8% of Flash’s remaining gap to the panel, so it is read against the baseline and is not a percentage. Five findings stand out.

Table 2: Main results: the seven multimodal entries, 18 tasks, sorted by O-EVAL. “Art.”: tasks with a scoreable artifact; cost: USD per 18-task pass. O-EVAL is the multi-reference panel score (%) and MULTI-BO the same score relative to the DeepSeek V4.1 Flash baseline (1.000 = level with Flash), both three-seed means (± std).

Model + harnessArt.O-EVAL (%)MULTI-BOCost (USD)
GPT-6 Astra + codex1864.58 ± 2.931.132 ± 0.02246.41
Fable 5.1 + claude-code1863.11 ± 1.331.081 ± 0.05272.80
DeepSeek-Flash + opencode1859.81 ± 1.251.0002.54
GPT-5.5 xhigh + codex1859.75 ± 1.550.998 ± 0.0199.37
Kimi K3 + opencode1858.23 ± 1.120.953 ± 0.0423.26
GLM-5.3 + opencode1857.78 ± 1.300.941 ± 0.0528.15
Qwen3.8 Flash + opencode1852.66 ± 1.170.803 ± 0.048.30

Table 2.

  1. The newest flagships lead. GPT-6 Astra + codex (64.58 ± 2.93) and Fable 5.1 + claude-code (63.11 ± 1.33) top the board, about 4.8 and 3.4 points above GPT-5.5 xhigh + codex (59.75 ± 1.55), and both clear the Flash baseline by 13.2% and 8.1% of the remaining gap respectively.

  1. Cost buys a real but bounded gap. The two leaders cost $246–$273 per run, while the cheapest strong entry, DeepSeek-Flash + opencode, reaches 59.81 ± 1.25 at $2.54—about 4.8 points behind the leader at roughly 1/100 the cost.

  1. Cheap open models cluster below the flagships. Among the low-cost entries DeepSeek-Flash (59.81), Kimi K3 (58.23), GLM-5.3 (57.78), and Qwen3.8 Flash (52.66) span about 7 points, and only the two flagships exceed the baseline index (1.132 and 1.081); DeepSeek-Flash defines it at 1.000 and every other entry falls below.

  1. A small gap is not yet a resolved ranking. Per-seed deviations reach 2.93 pp, so the ∼ 1.5 pp separation between the two leaders and between GPT-5.5 and Flash sits inside one standard deviation; we report the order but do not claim a significant difference between adjacent rows.

  1. More inference compute is not monotone. Running DeepSeek-Flash at its maximum reasoning tier drops its multi-reference score to 48.11% and its MULTI-BO to 0.864, well below the default tier’s 59.81 and 1.000, at higher cost: a larger deliberative budget does not simply buy accuracy on these tasks.

Hard gate and cost. The gate removes exactly the cells whose answers violate a stated constraint and leaves the ranking otherwise stable, so it is surgical rather than a blanket penalty. Cost is dominated by per-token price, not usage: GPT-5.5’s $99.37 is mostly cache-read input and output at roughly 33×/50× DeepSeek V4.1 Flash’s rates, despite comparable token counts.

Harness sensitivity. Each model is driven by an agent harness, so a natural question is whether the score reflects the model or the harness. Table 3 holds the model fixed and varies the harness, using DeepSeek-Flash and four harnesses. The score is essentially invariant: O-EVAL moves only 0.07 pp across all four (59.81–59.88), far inside the per-seed deviation, and the relative index moves in the same narrow band. A model’s own harness fits it marginally better—DeepSeek’s native harness gives the highest O-EVAL (59.88) and MULTI-BO (1.092)—but the difference is small. The board therefore measures the model far more than the harness it is wrapped in.

Table 3: Harness sensitivity, holding the model fixed (DeepSeek-Flash, 18 tasks, three seeds). Multi-BO is 1.000 for the opencode reference harness by definition.

HarnessO-EVAL (%)MULTI-BO
opencode59.81 ± 1.251.000
claude-code59.83 ± 1.131.081 ± 0.05
codex59.84 ± 1.211.085 ± 0.03
DeepSeek (native)59.88 ± 1.201.092 ± 0.03

Table 3.

Figure 3. Left: multi-reference O-EVAL versus cost (log scale)—the Pareto frontier is compressed, with near-frontier scores at two orders of magnitude lower cost. Centre: O-EVAL per entry. Right: O-EVAL by suite, showing the CUMCM–MCM gap.

Analysis

Per-task O-EVAL for four representative entries is in Appendix Table 6. Three patterns stand out. (a) The CUMCM–MCM gap is large and model-dependent. For the weaker entries, CUMCM cells sit near 100% while MCM cells sit between 0% and 60%: the engineering problems have well-defined final numbers most models reach, whereas MCM problems are more open and their anchors more contestable. The strongest entries narrow the gap only slightly (GPT-6 82.9 on CUMCM vs. 46.2 on MCM; GPT-5.5 79.7 vs. 39.8). A sizeable share of tasks is now solved near-perfectly—expected of a current benchmark—and the board’s discrimination comes from the open-ended MCM problems.

(b) Two tasks are all-or-nothing. On mcm-2024-b-submersible every entry scores 100%, while on mcm-2024-c-tennis even the flagships score below 40%. Such tasks either fall inside or outside the panel’s tolerance entirely, so they separate regimes of models but not models within one.

(c) Many low scores reflect reference scale/scope, not modeling error. Several tasks contain a metric whose model/panel ratio exceeds 5×5 \times: a host-fish index reported near 1080 by some papers while models give a normalised [0,1][0,1] quantity; a sustainability score near 1.007 against models reporting a 0–100 utility; a 6×66 \times6 grid against models discretising the same region into hundreds of cells; and a 72.67 → 73.52 band in which the paper’s own baseline scores zero.

Scoring pitfalls we found and fixed. An anchor-based benchmark is only as credible as its anchors, so we audited every scoring rule and anchor and report what the audit changed (Appendix § 14). Two gate false positives were removed without moving any score, and degenerate-band widening affected one cell (+0.35 pp). Excluding two non-comparable-unit metrics raised every entry by 0.05– 2.62 pp—the signature of a systematic scoring effect rather than a model difference. Most consequentially, the audit flagged the dragon-dance anchor as physically infeasible and we corrected it (§ 4.2).

What the feasibility layers caught. The gate hard-invalidated 7 cells, concentrated on heliostat (rated power below 60 MW, or a mirror below the 2 m minimum) plus single cells for smoke-screen, Juneau, and Olympic. The Lean layer turns two soft warnings into proofs and proves non-vacuity by accepting a corrected artifact under the predicate that rejects the reported one. On the one numerically well-posed task the runs agree to machine precision, and across the board the per-seed deviations stay small relative to the spread between entries (Table 2), so rankings do not depend on the seed.

From Findings to Model Development

TB-MathModel is meant to improve models, not only rank them. Three levers follow from the findings above, and each is trainable with signals TB-MathModel already produces.

Verified reward for RL. The feasible, oracle-normalised score is an executable, reproducible reward—every submission passes the gate and the Lean layer. Rejection-sampling (keep and rank gate-accepted trajectories by O-EVAL) works today; group-relative policy optimisation with the gate as a hard multiplier teaches the model to satisfy the physics before chasing the reference. The dragon-dance case motivates this: twelve models computed the right peak but were penalised, so what they lack is self-checking—exactly what a verifier in the reward provides.

Targets against the observed failures. Finding (a)—the open-ended MCM suite separates quality while CUMCM saturates—says to train assumptions, formulation, and sensitivity rather than closed-form solving; finding (c)—many losses are unit and scale mismatches—says to state and verify units, ranges, and schema before emitting a number. Both are rewarded directly by the artifact contract.

Feasibility the artifact can check. The dragon-dance audit exposes the failure a value-range check cannot: a consistent answer that is physically in-admissible because the artifact omits the evidence (no per-mirror layout, no peak-neighbourhood sample). Training models to emit that witness raises the score and makes them checkable, reaching level C.

Case studies. Three roles—a top proprietary agent, the oracle, and the low-cost open agent—are compared on three problems in Appendix § 13. On dragon-dance all three converge while the paper’s value is the infeasible outlier, so the models were right and our anchor was wrong. On crop-planting the paper’s constraints are jointly infeasible, so we anchor on a reproducible exact MILP optimum. On Juneau the sustainability score is reported on different unit scales, which is why the task schema, not the scorer, must pin a field’s range.

Conclusion

We presented TB-MathModel, the first agentic and verifiable terminal benchmark of 18 real mathematical-modeling problems, distinguished by direction-aware, one-sided scoring that never penalises beating the anchor, and two-layer feasibility verification combining a gate over six decidable invariant forms with 624 kernel-checked Lean obligations.

Scoring each task against a panel of human outstanding papers, the newest flagships lead across seven multimodal entries (GPT-6 Astra at O-EVAL 64.58%, Fable 5.1 at 63.11%), while the $2.54 open DeepSeek-Flash holds 59.81% at 1/100 the cost. We documented the scoring pitfalls we fixed, and showed a case where the benchmark itself was wrong and the models were right.

The two-layer design is the transferable idea: a fast gate covers every task and a proof assistant makes the audit’s own correctness machine-checkable, while stating what neither can answer. Because TB-MathModel gives a reproducible reward, its trajectories are also training material, and § 4.4 turns the findings into directions.

Acknowledgements

We thank OpenRefinery (https://openrefinery.ai/) for providing the LLM API access and evaluation support that enabled our experiments with GPT-6-Astra and Claude-Fable-5.1. In particular, we thank Kai Du and Ding Fan of OpenRefinery for their assistance. Their infrastructure and support were instrumental in conducting the model evaluations reported in this paper. Any opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of OpenRefinery.

Limitations

TB-MathModel normalises each task against a reproducible Outstanding-award solution, with per-anchor provenance so any reference can be re-derived; because modeling admits no single correct answer, the natural extension is a multi-paper panel reporting a spread over accepted solutions. The two verification layers decide every feasibility question a structured artifact can settle (unit/value ranges and internal consistency); external physical truth is a property of the world, so it is checked where a problem admits an independent replay, and any obligation whose evidence is missing is reported rather than silently passed. Panel size varies with the number of outstanding reports per problem (1 to 18 papers), and where humans themselves disagree—as on the tennis momentum test, where 10 of 11 papers reach the opposite conclusion from the anchor—we mark the task literature-split and keep it out of headline rankings. Scores are three-seed means, and the task shell carries over to engineering and cross-disciplinary problems.

Ethical Considerations

Statements, attachments, and figures are official CUMCM and MCM material used for research evaluation; we do not redistribute Outstanding-award solutions, and reproduction code derives from a public corpus with per-task provenance. Users should confirm the licence terms for their own redistribution. TB-MathModel measures and improves the modeling workflow of LLM agents rather than automating work presented as a team’s own, and entirely automated evaluation keeps the cost of broad participation low.

Use of Large Language Models

LLMs served as the systems under test—every entry in Table 2 is an LLM agent, which is the object of measurement—and as writing assistants that helped draft prose and check the internal consistency of LaTeX and numbers. Benchmark construction, scoring rules, feasibility checks, and Lean obligations were specified, implemented, and verified by the authors, and all numerical results are produced by the released scripts.

Appendix Overview

This supplementary material makes the main paper’s construction and claims auditable. Appendix § 8 gives the dataset composition, the per-task table, the delivered directory structure, and an explicit statement that TB-MathModel has no train/validation/test split. Appendix § 9 records the software and hardware environment and the network isolation policy. Appendix § 10 lists the per-entry configuration, the agent command templates, and the retry/artifact-recovery details. Appendix § 11 disambiguates the three objects called “baseline”. Appendix § 12 reproduces the prompt template and the harness-wrapping layer. Appendix § 13 gives the full numerics behind the three case studies. Appendix § 14 gives the scoring formulas, the excluded metrics, and the hard-gate decisions. Appendix § 15 catalogues the Lean obligations, the trust audit, and the discrimination suite.

  • Appendix § 8: dataset composition, per-task table, delivered structure, and split

  • Appendix § 9: host, agent container, verifier container, resource limits, network isolation

  • Appendix § 10: per-entry configuration, command templates, retries, cost accounting

  • Appendix § 11: flash baseline, Outstanding anchor, legacy endpoints, anchor caveats

  • Appendix § 12: prompt structure, artifact block, harness wrapping, version caveat

  • Appendix § 13: full numeric case-study tables

  • Appendix § 14: metric panel, formulas, excluded metrics, hard gates

  • Appendix § 15: obligation counts, representative obligations, trust audit, discrimination suite

Dataset Details

Composition

TB-MathModel has 18 tasks, listed in Table 4 (main text). The composition is CUMCM (Chinese) 2023, 2024, 2025 ×\times problems A, B, C = 9 tasks, plus MCM (English) 2023, 2024, 2025 ×\times problems A, B, C = 9 tasks. Each task is a full contest problem with several sub-questions; only the numeric core of the final question is scored. Problem A is typically physical/engineering, B optimization/data, and C data-science, so taking all three of each year avoids topical cherry-picking.

SlugContestTopicFinal Q
cumcm-2023-a-heliostatCUMCM 2023 AHeliostat field designQ3
cumcm-2023-b-multibeamCUMCM 2023 BMultibeam survey linesQ4
cumcm-2023-c-vegetableCUMCM 2023 CVegetable pricing and replenishmentQ4
cumcm-2024-a-dragonCUMCM 2024 ABench-dragon kinematicsQ5
cumcm-2024-b-productionCUMCM 2024 BProduction decisionQ4
cumcm-2024-c-cropCUMCM 2024 CCrop-planting strategyQ1
cumcm-2025-a-smokeCUMCM 2025 AUAV smoke-screen strategyQ5
cumcm-2025-b-sicCUMCM 2025 BSiC epi-layer thicknessQ3
cumcm-2025-c-niptCUMCM 2025 CNIPT timing / abnormalityQ4
mcm-2023-a-plantMCM 2023 ADrought plant communitiesfinal
mcm-2023-b-maasaiMCM 2023 BMaasai Mara land usefinal
mcm-2023-c-wordleMCM 2023 CPredicting Wordle resultsfinal
mcm-2024-a-lampreyMCM 2024 ASex ratios / resource supplyfinal
mcm-2024-b-submersibleMCM 2024 BSubmersible searchQ4
mcm-2024-c-tennisMCM 2024 CMomentum in tennisQ5
mcm-2025-a-stairMCM 2025 AStair wear / footfallQ8
mcm-2025-b-juneauMCM 2025 BSustainable tourism (Juneau)Q3
mcm-2025-c-olympicMCM 2025 COlympic medal modelsfinal

Table 4. The 18 TB-MathModel tasks and their O-award anchors. “Final Q” is the contest question whose numeric core is scored; oracle coverage is 18/18.

An official statement figure showing a solar panel installation

Figure 4. An official statement figure, cropped from the CUMCM 2023 A PDF and delivered inline in the prompt. 21 such figures across 6 tasks are load-bearing: the final answer depends on reading them.

Per-task summary

Table 5 summarises each task. “#Fig.” counts unique figures cropped from the official PDF; “#Metric” is the number of scored metrics.

SlugFinal QSource#Fig.#Metric
cumcm-2023-a-heliostatQ3CUMCM 2023 A25
cumcm-2023-b-multibeamQ4CUMCM 2023 B84
cumcm-2023-c-vegetableQ4CUMCM 2023 C02
cumcm-2024-a-dragonQ5CUMCM 2024 A52
cumcm-2024-b-productionQ4CUMCM 2024 B16
cumcm-2024-c-cropQ1CUMCM 2024 C02
cumcm-2025-a-smokeQ5CUMCM 2025 A02
cumcm-2025-b-sicQ3CUMCM 2025 B22
cumcm-2025-c-niptQ4CUMCM 2025 C02
mcm-2023-a-plantfinalMCM 2023 A03
mcm-2023-b-maasaifinalMCM 2023 B05
mcm-2023-c-wordlefinalMCM 2023 C22
mcm-2024-a-lampreyfinalMCM 2024 A03
mcm-2024-b-submersibleQ4MCM 2024 B02
mcm-2024-c-tennisQ5MCM 2024 C04
mcm-2025-a-stairQ8MCM 2025 A13
mcm-2025-b-juneauQ3MCM 2025 B06
mcm-2025-c-olympicfinalMCM 2025 C05
Total2160†

Table 5. Per-task summary. “Final Q” is the sanctioned final question.

†60 scored metrics across the 18 tasks; one further Wordle metric is declared at weight 0 and excluded from scoring. The counts are verified against the released tasks; the 21 figures are unique after de-duplication and are delivered twice—once inline in the instruction file and once under the container figure path. Data sources are the official contest material: CUMCM statements under cumcm/problems/{year}/{X}.md and MCM under mcm/problems/{year}/{X}.md; Outstanding-paper PDFs/OCR used for anchors come from the same corpus’s outstanding/ tree, and every task records its paper identifier, the PDF/OCR source paths, and its reproduction_scope.

Per-task multi-reference scores

Delivered structure

Each task directory contains an instruction file (the prompt), a metadata file (limits), an environment directory (agent image plus data), a tests directory (verifier, scoring config, anchor), and a solution directory (oracle). Inside the environment data directory are the statement copy, the output schema (field names and types only), a source manifest mapping each visible file to its corpus path, the verbatim attachments under flat filenames with an index, a full-path mirror of the original corpus, and the cropped figures for the six tasks that have them.

Split

There is no train/validation/test split. TB-MathModel is a pure evaluation set: the 18 tasks are all evaluation tasks, evaluated uniformly, with no held-out partition and no per-task difficulty tier. (One task’s oracle model contains an in-task train/test split for the Wordle problem; that is part of that problem’s modeling content and is unrelated to benchmark partitioning.)

Software and Hardware Environment

Host and framework

The evaluation harness runs from a macOS host (arm64). It orchestrates per-task Docker containers; the enforced resource limits are container limits, not host limits.

Agent container

All 18 tasks use the same base and dependency set. The base image is python:3.12-slim plus uv. The Python environment at /opt/venv pins NumPy 2.0.2, pandas 2.2.2, SciPy 1.14.1, scikit-learn 1.5.2, Matplotlib 3.9.2 (MPLBACKEND=Agg), openpyxl 3.1.5, and xlrd 2.0.1. Writable paths are /root/data (read-only inputs), /root/results (the scored artifact), and /logs/agent.

Verifier container

The verifier runs in a separate container built from a minimal python:3.12-slim image installing only pytest. It has no network and no access to the agent’s working directory beyond the copied artifacts. The verifier timeout is 180 s.

Resource limits

The limits are identical for all tasks: 2 CPUs, 8192 MB memory, 20480 MB storage, 0 GPUs. The build timeout is 900 s and the agent timeout 14400 s (4 h), also stated at the end of each prompt. The recorded runs used one concurrent trial and no retries.

Network isolation

Every task disables internet access. Model traffic is routed through a local retry proxy, so an agent can call the model API but cannot fetch solutions, download packages, or install tooling at run time. Because installation is impossible after a trial starts, CLI agents (codex, opencode, claude-code) must be present in the image; the third-party operator’s runs pre-installed the CLIs at the end of the Dockerfile and allow-listed only the model gateway, leaving the statement, data, and verifier unchanged.

Frozen versions

For reproducibility we pin the harness version, the Python dependency set above, and the local pricing snapshot used for cost, so that the reported costs are stable across reruns.

Experimental Settings

Per-entry configuration and results

Table 7 lists the seven multimodal entries with the harness and reasoning tier actually passed and their full scores. Multimodal status means the harness exposes an image channel. Every entry is run with three seeds per task.

ModelHarness / tierArt.O-EVAL(%)MULTI-BO
GPT-6 Astracodex CLI / max1864.58±2.9364.58 \pm2.931.132±0.021.132 \pm0.02
Fable 5.1claude-code CLI / max1863.11±1.3363.11 \pm1.331.081±0.051.081 \pm0.05
GPT-5.5 xhighcodex CLI / xhigh1859.75±1.5559.75 \pm1.550.998±0.010.998 \pm0.01
DeepSeek-Flashopencode CLI / high1859.81±1.2559.81 \pm1.251.000
Kimi K3opencode CLI / high1858.23±1.1258.23 \pm1.120.953±0.040.953 \pm0.04
GLM-5.3opencode CLI / high1857.78±1.3057.78 \pm1.300.941±0.050.941 \pm0.05
Qwen3.8 Flashopencode CLI / high1852.66±1.1752.66 \pm1.170.803±0.040.803 \pm0.04

Table 7. Multimodal board, 18 tasks, sorted by O-EVAL. “Art.” is the number of tasks with a scoreable artifact; O-EVAL is the multi-reference panel score and MULTI-BO the same score relative to the DeepSeek V4.1 Flash baseline, both three-seed means (±\pm std).

Agent command templates

The task statement is passed to the agent as its prompt. Representative commands from the recorded trials:

# codex
codex exec --dangerously-bypass-approvals-and-sandbox --skipgit-repo-check \
--model gpt-5.5 --json --enable unified_exec \
-c model_reasoning_effort=xhigh -- '<instruction.md body>'
# opencode
opencode --model=openai/deepseek-flash run --format=json --
thinking \
--dangerously-skip-permissions -- '<instruction.md body>'

Each CLI harness receives the statement as its positional prompt; the harness prepends its own generic system prompt before it.

Retries, concurrency, and artifact recovery

All entries used one concurrent trial and no automatic retries; each of the three seeds is an independent pass over the 18 tasks. Two subtleties are disclosed. First, misfiled artifacts: two tasks wrote the result with a hyphen instead of the required underscore, so the harness did not collect it; we replayed the recorded run’s scripts to reproduce the artifact and re-scored it. Second, third-party runs: the GPT-6 Astra and Fable 5.1 entries were run by independent operators; because their verifier used a different reward formula, we re-scored their saved artifacts with our scorer and current anchors so that all entries share one scoring definition. Where an artifact was not packaged, we reconstructed the scored fields from the saved per-metric records, verified to reproduce the operator’s own numbers exactly on the tasks where the full artifact was present.

Cost accounting

Cost is computed from measured token usage against a fixed price snapshot, with two conservative adjustments stated in the report: reasoning tokens that a provider bills outside its completion counter are added to output, and where a provider charges a peak/off-peak rate, the peak rate is used. For the two third-party entries, cost is their own measured per-run USD. Because the pricing file is local and fixed, costs do not drift between reruns.

Baseline Information

The word “baseline” is overloaded in this benchmark; we distinguish three distinct objects so that the numbers are unambiguous.

The flash baseline (score normalisation)

Robust BO-Eval is computed against a single reference submission, the DeepSeek V4 Flash baseline run on the same tasks. It defines the zero point:

TaskGPT-6 AstraFable 5.1GPT-5.5 xhighDeepSeek-Flash
cumcm-2023-a-heliostat86.280.780.580.2
cumcm-2023-b-multibeam94.080.591.759.3
cumcm-2023-c-vegetable64.364.364.364.3
cumcm-2024-a-dragon83.483.483.483.4
cumcm-2024-b-production95.780.587.287.2
cumcm-2024-c-crop88.283.599.763.8
cumcm-2025-a-smoke100.0100.0100.0100.0
cumcm-2025-b-sic88.286.780.378.7
cumcm-2025-c46.477.029.776.4
mcm-2023-a-plant21.324.627.727.7
mcm-2023-b-maasai34.535.143.143.1
mcm-2023-c46.149.640.956.4
mcm-2024-a47.143.810.760.9
mcm-2024-b-submersible100.0100.0100.0100.0
mcm-2024-c-tennis36.436.49.19.1
mcm-2025-a-stair21.413.112.912.8
mcm-2025-b-juneau42.942.942.942.9
mcm-2025-c-olympic66.454.171.230.1
Average64.5863.1159.7559.81

Table 6. Per-task multi-reference O-EVAL (%) for four representative entries. Each column’s mean equals that entry’s main-table O-EVAL, so the per-task values reconcile with the board.

ROBUST BO-EVAL normalises the model–baseline gain by the oracle–baseline gap. It is a model submission, not a curated baseline panel. and surfaced in the per-task tables.

The Outstanding anchor (score ceiling)

Each task’s ceiling is the reproduction of one Outstanding-award paper, which is the value that scores O-EVAL = 100%. Table 8 lists the anchor paper identifier for each task. Each anchor is a runnable script that reads the official attachments and emits the artifact JSON, plus a committed expected result recording its values; coverage is 18/18.

SlugPaper idSlugPaper id
cumcm-2023-a-heliostatA175mcm-2023-a-plant2309229
cumcm-2023-b-multibeamB226mcm-2023-b-maasai2315379
cumcm-2023-c-vegetableC050mcm-2023-c-wordle2307946
cumcm-2024-a-dragonA016mcm-2024-a-lamprey2407093
cumcm-2024-b-productionB159mcm-2024-b-submersible2419984
cumcm-2024-c-cropC038mcm-2024-c-tennis2401298
cumcm-2025-a-smokeA196mcm-2025-a-stair2501909
cumcm-2025-b-sicB157mcm-2025-b-juneau2504448
cumcm-2025-c-niptC023mcm-2025-c-olympic2505964

Table 8. Outstanding-award anchor per task.

Legacy per-task baseline endpoints

Fifteen of the 18 tasks use the legacy target-distance-to-Outstanding mode with a baseline endpoint; three (Wordle, stair-wear, and Juneau tourism) use true per-metric baseline-to-Outstanding min–max endpoints. The distinction is recorded per metric in each scoring configuration

Anchor caveats

Two anchors carry caveats that affect how their scores should be read. Crop-planting is a marked special case: the paper’s full constraint set is jointly infeasible on the official attachments and its reported value came from an undisclosed heuristic, so the anchor is a reproducible exact MILP optimum under the paper’s per-plot demand cap. Dragon-dance was corrected: the paper’s 1.406276 m/s is infeasible (a handle would reach 2.2568 m/s), so the anchor is the continuous-time value 1.246266 m/s and the metric direction is target_value. Both are examples of the general principle that an anchor is a reproduction of one team’s choices, and that its provenance must be inspectable.

Prompt Design

The prompt is the task statement

There is no hand-written prompt wrapper in this benchmark. The prompt is the task’s instruction file, delivered verbatim by the harness. Its structure is fixed across tasks: a machine-readable canary comment (stripped before delivery); an “Original Contest Problem” section containing the verbatim statement with each cited figure embedded inline; any results tables the statement requires, as plain text; a note listing the in-container figure paths; a “Required Benchmark Artifact” section; and the time-limit line.

The artifact block (verbatim template)

# Required Benchmark Artifact

Solve the contest problem above. The official attachments are
mirrored at
`/root/data/attachments` (flat filenames; see
`/root/data/attachments/INDEX.md`) and also remain at their full corpus paths
under `/root/data/repo`. Use them whenever they are relevant.
Write your final
structured answer to:

/root/results/<slug>_result.json

The result must be valid JSON and should follow the shape
shown in
`/root/data/output_schema.json`. Include the numerical
quantities, modeling
decisions, diagnostics, sensitivity or uncertainty results,
conclusions for the
contest questions, and generated artifact paths needed to
support your answer.
You may also write supplementary CSV, XLSX, or PNG files under
`/root/results`,
but the JSON file above is the required artifact.

You have 14400 seconds to complete this task. Do not cheat by
using online
solutions or hints specific to this task.

Harness wrapping

The harnesses deliver the same text differently. Each CLI harness prepends its own generic command-line system prompt and then supplies the statement as the positional prompt.

Version caveat

The instruction files were revised after some trials were recorded: a figure-path section was added, and one MCM statement was re-sourced from the official PDF. The prompt actually delivered in a given trial is the one captured in that trial’s log; for reproducibility, cite those, not the current file. This is why we describe the prompt structure above rather than quoting today’s file as if it were what every model saw.

What the prompt withholds by design

By design the prompt is a pure task description: no modeling hints, no method suggestions, and no pointer to the scored fields beyond the schema. It leaves the agent to infer which numbers matter, which keeps the setting a test of the whole modeling problem rather than a fill-in-the-schema exercise. This keeps the benchmark a test of doing the whole modeling problem, at the cost of the agent having to infer which numbers matter.

Case Studies (Full Numerics)

We give the full per-field comparison for the three case studies. Values are read from each role’s saved artifact; the oracle column is the anchor. All three tables span both columns.

The reconstruction of the true peak is a continuous-time maximum of the handle-speed ratio; the paper sampled integer seconds and stepped over it. The exact identity head_speed×ratio=2head\_speed \times ratio = 2 m/s, checked by both the programmatic gate and the Lean layer, is what exposes the paper value as infeasible: the paper’s isolated value 1.406276 m/s (with ratio 1.422195) implies 2.2568 m/s at a handle.

The paper’s full constraint set is infeasible on the official attachments, so the anchor is a reproducible MILP optimum and the models’ plans sit in the same regime. The anchor lies between the two models on the discount column, which is why a single scalar cannot summarise this task.

QuantityOracle (anchor)GPT-5.5DeepSeek V4.1
max_head_speed_mps1.2462661.246266361.24626636
max_speed_ratio1.6047931.604793381.60479338

Table 9. Dragon-dance (CUMCM 2024 A, Q5). All three roles agree; the paper value is the outlier and is infeasible.

Quantity (yuan, 7-year)Anchor (exact MILP)GPT-5.5DeepSeek V4.1
waste-surplus profit40,418,87740,320,58741,559,515
50%-discount profit58,718,96766,485,66270,378,276

Table 10. Crop-planting (CUMCM 2024 C, Q1). Special case: the anchor is a reproducible exact optimum under the paper’s per-plot cap.

This is the clearest scale problem. The anchor is on a ∼1\sim1 unit scale; one third-party agent reports the same field on a 0–100 “multiattribute utility” scale that it explicitly declares in its own artifact. Because the task’s output schema does not pin the range, the agent is not wrong—which is why the correct repair is to specify the field’s range in the schema, not to rescale in the scorer.

Discussion

Dragon-dance. The paper’s isolated value (1.406276) is the wrong one. This case matters most because it inverts the usual story: the models were right and the reference was wrong, and the fix was to the benchmark. It also shows why the feasibility layer is not redundant with scoring—“ratio × head speed = 2 m/s” is an exact identity that both layers check, and it is what made the paper value’s infeasibility visible.

Crop-planting. GPT-5.5 and DeepSeek produce plan profits in the same regime, and the optimum sits between them on one column—which is why a single scalar cannot summarise the task. When the human paper is not reproducible, the honest move is to anchor on a reproducible value and mark it, not to penalise models for matching an irreproducible number.

Juneau. The oracle recommends an 11,000/day cap at 55 USD with a sustainability score of 1.007, GPT-5.5 recommends 20,000/day at 20 USD on its own 0–1 scale, and DeepSeek recommends 16,000/day at 30 USD. This argues that the task schema—not just the scorer—must state a field’s range and units, which is a fix to the task rather than to the code.

Scoring Details

The metric panel

Every task ships a scoring configuration declaring its scored metrics. A metric entry specifies a path into the artifact, an oracle value, optional baseline and outstanding values, a direction (with a matching semantic direction), a normalisation mode, and a weight. Across the 18 tasks there are 60 scored metrics; one further Wordle metric is declared at weight 0 and recorded but excluded from the panel. The per-task counts are in Table 5.

Formulas

Let RR be a submission’s weighted panel raw, FF the DeepSeek V4 Flash baseline’s, OO the oracle’s, and gap⁡=O−F\operatorname{gap}=O-F. We report O-EVAL =clamp⁡(R/O,0,1)=\operatorname{clamp}(R/O,0,1) and a gap-compensated Ro-BUST BO-EVAL based on the raw gain R−FR-F:

clip⁡(R−Fgap⁡,±1)if gap≥0.10,clip⁡(R−F,±0.10)otherwise,\begin{aligned} \operatorname{clip}\left(\frac{R-F}{\operatorname{gap}},\pm1\right) &\quad\text{if gap} \ge0.10,\\ \operatorname{clip}(R-F,\pm0.10) &\quad\text{otherwise,} \end{aligned}

where ±\pm denotes the symmetric bound. A missing or non-scoreable artifact gets RoBUST BO-EVAL =−1=-1. The per-metric reward is given in § 3.1; the one-sided property (meeting or beating the anchor never scores below 1.0) is locked by a contract test.

Excluded metrics

Two metrics are excluded from the panel for every model and the oracle, both by configuration-only rules: the Olympic fourth recommendation (the prompt asks for three countries but the reproduction emits four rows), and the two lamprey raw population indices when the anchor magnitude exceeds 5 (the anchor is an arbitrary raw index with no schema-pinned unit). Out-of-domain values

QuantityAnchorGPT-5.5DeepSeek V4.1Fable 5.1
daily cap11,00020,00016,00014,000
visitor fee (USD)55203030
annual visitors (M)1.4921.4531.4341.728
resident acceptance1.0000.7430.9380.696
sustainability score1.0070.9000.70679.1
total revenue ($M) | 431.6 | 369.6 | 366.7 | 458.5

Table 11. Juneau tourism (MCM 2025 B, final question). The sustainability score is reported on different unit scales by different agents.

FormExample obligation
domain / rangeprobability ∈[0,1]\in[0,1]; thickness >0> 0
integer / binarydecision bits ∈{0,1}\in\{0,1\}; counts integer
orderingq95≥q_{95} \geq mean; peak ≥\geq average
exact identityarea = count ×\times w ×\times h; union ≤∑\leq\sum
finite domaindecision-bit count ∈{16,17}\in\{16,17\}
partition / sumland-use counts sum to the grid

Table 12. The six decidable invariant forms covering the hard-error branches of all 18 tasks.

(correlations outside [−1,1][-1,1], unit-interval quantities outside [0,1][0,1], negative counts, non-positive physical quantities) are not excluded but have their reward set to zero, with the reason recorded.

Hard-gate decisions

Seven whole-task penalties were applied across the entries: four heliostat violations (rated power below 60 MW, or a mirror below the 2 m minimum), one smoke-screen plan failing an independent coverage replay, one Juneau answer with unit-invalid acceptance/sustainability values, and one Olympic answer with a degenerate coach-effect result.

Pitfalls found and fixed

An anchor-based benchmark is only as credible as its anchors, so we audited the scoring rules and anchors and report what the audit changed. Two gate false positives (a signed elasticity read as a negative count; a missing field read as a hard contradiction) were corrected but changed no score, and degenerate-band widening moved one score (+0.35 pp for v4 pro). Non-comparable-unit exclusion raised every entry by 0.05–2.62 pp, since the penalty applied to all submissions equally—the signature of a systematic scoring error rather than a model difference. Finally, the dragon-dance anchor was itself physically infeasible: the paper’s 1.406276 m/s would drive a handle to 1.406276 × 1.604793 = 2.2568 m/s, above the 2 m/s cap, because it maximised over integer seconds and stepped over the true continuous-time peak at t=14.48t = 14.48 s. We corrected the anchor to 1.246266 m/s; twelve of the evaluated models had reported the correct value and had been sitting at 49.82%, while the two matching the erroneous paper value had been at 100%. Here the benchmark, not the model, was wrong (Appendices § 14 and § 15).

Degenerate-band and validity rules

The band-widening rule applies to the endpoint metrics of the three endpoint-style tasks. Applying all scoring fixes in this paper moved every entry upward by 0.05–2.62 pp, because the affected penalties applied to all submissions equally—the diagnostic signature of a systematic scoring error rather than a model difference.

Lean Obligation Catalogue

Per-task obligation counts

The generator emits one obligation per checkable field; the totals by task are in Table 14.

TaskOblig.SkippedTaskOblig.
cumcm-2023-a-heliostat120mcm-2023-a-plant66
cumcm-2023-b-multibeam31mcm-2023-b-maasai17
cumcm-2023-c-vegetable70mcm-2023-c-wordle3
cumcm-2024-a-dragon30mcm-2024-a-lamprey302
cumcm-2024-b-production610mcm-2024-b-submersible8
cumcm-2024-c-crop50mcm-2024-c-tennis7
cumcm-2025-a-smoke100mcm-2025-a-stair6
cumcm-2025-b-sic40mcm-2025-b-juneau12
cumcm-2025-c-nipt160mcm-2025-c-olympic82

Table 14. Obligations per task (624 total across 18 tasks). “Skipped” obligations were reported as missing evidence rather than silently satisfied.

Four skipped obligations are visible in the generator output: one missing overlap field (multibeam), scenarios whose declared grid count is null (Maasai partition), and one recommendation-count interpretation difference (Olympic).

Representative obligations

Over the invariant library, a task supplies bounds and the obligations are generated. Examples:

-- domain/range: a rate must lie in [0,1] (scaled ×1000) example : (0 <= 43 /\ 43 <= 1000) := by decide

-- ordering: q95 must not fall below the mean example : (posterior_mean <= posterior_q95) := by ...

-- exact identity: head_speed * ratio = 2 m/s (scaled) example : (1980000 <= 1246266 * 1604793 /\ 1246266 * 1604793 <= 2020000) := by decide

-- finite domain: count of decision bits is 16 or 17 example : (len = 16 \/ len = 17) := by decide

Every generated obligation is annotated with the JSON path it came from, so a reviewer can trace each claim back to the artifact.

LevelContentLayer
A — unit / rangeprobability ∈[0,1]\in[0, 1], thickness >0> 0gate + Lean
B — consistencyarea = count ×\times w ×\times h; partitions sumgate + Lean
C — physical truthdoes this field really produce 60 MW?neither

Table 13. Three levels of feasibility. Both layers cover A and B; C needs formalised physics.

Trust audit (condensed excerpt)

'Feasibility.atMost_trans'     axioms: propext, Quot.sound
'Feasibility.atLeast_mono'     axioms: propext, Quot.sound
'CropPlanting.ua_feasible'     no axioms
'CropPlanting.ua_in_interpretation_gap'     no axioms
'Juneau.ua_acceptance_percent_only'     no axioms
'Juneau.ua_sustainability_out_for_any_sane_tol'
                              axioms: propext, Quot.sound
'JuneauSensitivity.reported_infeasible'
                              axioms: propext, Quot.sound

Only propext and Quot.sound (Lean’s standard classical logic) appear, or no axioms at all. There is no sorryAx and no Lean.ofReduceBool (i.e. no native_decide); the pipeline greps for both.

Discrimination suite

The discrimination script mutates one numeric leaf at a time and requires the build to fail and to cite the mutated field. Ten mutations across the six forms are covered (out-of-range probability, out-of-range correlation, reversed ordering, violated exact identity, non-positive thickness, reversed interval, non-binary decision bit, negative acceptance index, q95q_{95} below mean, zero species count). All ten are rejected; the suite exits non-zero if any is not.

Reproducing the layer

Toolchain is Lean 4 via elan, with no Mathlib; a clean build completes in about a second. The generator also runs against the committed Outstanding-award references, so the layer can be exercised without the saved agent artifacts.

References

  1. [1]Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, and 1 others. 2024. InfAgent-DA-Bench: Evaluating agents on data analysis tasks. arXiv preprint arXiv:2401.05507.
  2. [2]Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. 2024. OptiMUS: Scalable optimization modeling with (MILP) solvers and large language models. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 577–596. PMLR.
  3. [3]Jun Sh ern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Alexander Mądry. 2025. MLE-bench: Evaluating machine learning agents on machine learning engineering. Preprint, arXiv:2410.07095. The Thirteenth International Conference on Learning Representations (ICLR), Oral.arxiv.org/abs/2410.07095
  4. [4]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reïchiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.arxiv.org/abs/2110.14168
  5. [5]Leonardo de Moura and Sebastian Ullrich. 2021. The Lean 4 theorem prover and programming language. In Automated Deduction – CADE 28, Lecture Notes in Computer Science, pages 625–635. Springer.DOI
  6. [7]Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily Oliveira Santos, and 1 others. 2024. FrontierMath: A benchmark for evaluating advanced mathematical reasoning in ai. Preprint, arXiv:2411.04872.arxiv.org/abs/2411.04872
  7. [8]Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, and 1 others. 2024. BLADE: Benchmarking language model agents for data-driven science. arXiv preprint arXiv:2408.09667.arxiv.org/abs/2408.09667
  8. [9]Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. 2024. DS-Agent: Automated data science by empowering large language models with case-based reasoning. In Proceedings of the 41st International Conference on Machine Learning (ICML).arxiv.org/abs/2402.17453
  9. [10]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems (NeurIPS).
  10. [11]Chenyu Huang, Zhengyang Tang, Shixi Hu, Ruoqing Jiang, Xin Zheng, Dongdong Ge, Benyou Wang, and Zizhuo Wang. 2025a. ORLM: A customizable framework in training large models for automated optimization modeling. Operations Research, 73(6):2986–3009.DOI
  11. [12]Xuhan Huang, Qingning Shen, Yan Hu, Anningzhe Gao, and Benyou Wang. 2025b. LLMs for mathematical modeling: Towards bridging the gap between natural and mathematical languages. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2678–2710.arxiv.org/abs/2405.13144
  12. [13]Thomas Hubert, Rishi Mehta, Laurent Sartran, Miklós Z. Horváth, Goran Žužić, Eric Wieser, Aja Huang, Julian Schtrittwieser, and 1 others. 2025. Olympiad-level formal mathematical reasoning with reinforcement learning. Nature, 651:607–613.DOI
  13. [14]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations (ICLR).arxiv.org/abs/2310.06770
  14. [15]Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tianyu Zheng, Xinyao Niu, Xiang Yue, Yue Wang, Jian Yang, Jiacheng Liu, and 1 others. 2024. AutoKaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20244.
  15. [16]Vinicius Lima, Dzung T. Phan, Jayant Kalagnanam, Dhaval Patel, and Nianjun Zhou. 2025. Toward a trustworthy optimization modeling agent via verifiable synthetic data generation. Preprint, arXiv:2508.03117.arxiv.org/abs/2508.03117
  16. [17]Fan Liu, Zherui Yang, Cancheng Liu, Tianrui Song, Xiaofeng Gao, and Hao Liu. 2025. MM-Agent: LLM as agents for real-world mathematical modeling problem. Preprint, arXiv:2505.14148.arxiv.org/abs/2505.14148
  17. [19]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Ludwig Schmid, and 1 others. 2026. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. Preprint, arXiv:2601.11868. Project page: https://www.tbench.ai/.arxiv.org/abs/2601.11868
  18. [20]Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: a benchmark for general assistants. In The Twelfth International Conference on Learning Representations (ICLR).arxiv.org/abs/2311.12983

Paper details

vlnc:W_y39fnv69
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLMs

View statement
Looking for related papers…
Contents