WorksComputer Science
JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams
Abstract
Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.
Code: github.com/SakanaAI/JFinBench
Data: huggingface.co/datasets/SakanaAI/JFinBench
Introduction
Large language models (LLMs) increasingly take on professional work ([28]), including software engineering ([24]), web browsing ([54]), enterprise workflows ([42, 50]), economically valuable occupational tasks (Patwardhan et al., 2026), and even running a business [5, 52]. Finance is a critical domain for LLM applications [61], spanning everyday tasks such as answering tax and investment questions as well as more complex tasks requiring specialized financial knowledge [6, 53]. Accurately assessing these capabilities is essential for advancing the application of LLMs in finance.
However, existing financial benchmarks have two important limitations. First, they provide limited coverage of country-specific financial knowledge. Financial expertise depends on local tax rules, accounting standards, and regulations, yet most existing benchmarks [49, 57] focus on English-language data and U.S.-centric settings. For Japanese finance, Hirano [21] built a benchmark from certification exams, but it covers four exams with multiple-choice questions only, leaving free-form answers such as tax calculations and essays, as well as questions that depend on figures, unevaluated. Second, existing benchmarks are difficult to keep up to date. Tax rules and regulations are revised frequently, so questions from past exam editions may test rules that no longer apply. Moreover, once released, benchmark questions can leak into model training data, so evaluation scores may increasingly reflect memorization rather than generalization (Sainz et al., 2023). Although regularly incorporating newly released material can mitigate these issues [19, 23, 55], converting new source documents into reliable structured examples suitable for model evaluation requires substantial effort, hindering frequent updates.
We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six financial certification exams. These exams cover diverse areas of Japanese finance and its regulations, from accounting and auditing to tax law, financial planning, and banking. They are also administered regularly, so each new edition supplies fresh expert-written questions that postdate the training data of existing models. To convert the exam PDFs into structured examples suitable for model evaluation, we develop the human-in-the-loop agentic pipeline shown in Figure 1. In this pipeline, a coding agent generates exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. Because the agent tailors the code to each exam, the pipeline adapts to diverse question formats, including multiple choice, essays, and image-based questions. This reduces the effort required to incorporate new exam editions, making it practical to refresh the benchmark over time.
Figure 1. Human-in-the-loop agentic construction pipeline of JFinBench. Generic scripts transcribe each exam PDF into layout-preserving markdown with a VLM, code written by a coding agent for each exam turns the transcripts into structured examples, and a human verifies every example side by side against the source page and reports problems back to the agent, which fixes its code.
Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall and approach saturation on several exams. In contrast, the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points, suggesting substantial room for improvement in open-weight models’ specialized Japanese financial capabilities. This gap stands in contrast to agentic coding, where Kimi K3.1 Pro is reported to outperform GPT-6 Astra on Terminal-Bench [31, 34], suggesting that open-weight models have nearly caught up in domains that model developers prioritize but not in specialized domains such as Japanese finance. Our failure analysis indicates that most errors arise from missing knowledge, while calculation errors are rare. With terminal and web access, most models improve by retrieving missing knowledge from the web. The gap narrows but approximately 4 percentage points remain, and the ranking of models changes little (Spearman $\rho = 0.87$), suggesting that knowledge held in the model’s parameters remains valuable even with tools.
Related Work
Financial benchmarks. Financial benchmarks for LLMs have progressed from relatively simple tasks, such as numerical reasoning over corporate filings in FinQA [8], toward more practical and complex ones. The Finance Agent Benchmark has agents research recent SEC filings with web search and EDGAR access [6], and FrontierFinance evaluates the full workflow of professional investment research ([61]). These resources primarily focus on English-language financial data and U.S.-centric settings, leaving country-specific financial systems less well covered.
For Japanese finance, where LLM applications are an active area of study ([35, 53]), a few benchmarks have begun to address this gap by drawing on certification exams. Masuda et al. [29] evaluated ChatGPT on the auditing section of the CPA short-answer exam, and Hirano [21] pioneered benchmark construction from Japanese financial certification exams, covering four exams. JFinBench extends this line of work in three directions. First, it draws on the newest exam editions, held in 2026, to limit contamination, whereas prior benchmarks use editions from 2013–2023 and undated sample questions. Second, it covers question formats beyond multiple choice, including image-dependent and free-form questions. Third, it comes with a pipeline that keeps the benchmark refreshable as new editions are released. Table 1 summarizes the comparison.
Table 1. Exam-based Japanese financial benchmarks. MC: multiple choice. Free: free-response, form, and essay answers graded with rubrics. VQA: image-dependent questions retained as visual inputs.
Exam-based benchmarks. Examinations provide a natural source of evaluation data because they are authored by domain experts and designed to distinguish different levels of human competence: MMLU ([20]) draws on school-level examinations and standardized admission tests. More recent studies have turned to professional licensing examinations, including the bar exam ([10, 22, 26] (OpenAI 2024a)) and medical licensing exams ([25, 36, 58]), and the mathematics competition AIME ([30]) has become a standard benchmark for reasoning models. JFinBench follows this line of work by using Japanese financial certification exams as a source of expert-authored evaluation material.
Live benchmarks. A persistent challenge for benchmark-based evaluation is data contamination: benchmark questions may enter model training data, so scores come to reflect memorization rather than generalization ((Sainz et al., 2023)). Post-hoc detection methods exist ([14, 43]), but a more fundamental remedy is to evaluate on material that postdates the training data, as in live benchmarks such as LiveCodeBench, which periodically incorporate newly released problems ([19, 23, 55]). JFinBench brings this live-benchmark approach to Japanese finance, using the certification exams, which are held regularly and published officially, as a recurring source of fresh, expert-authored evaluation material.
Scalable benchmark construction. Building a benchmark by hand is laborious ([31]), so a growing line of work uses programs and agents to make its construction more efficient and scalable. One approach mines existing artifacts and converts them automatically: SWE-bench ([24]) builds tasks from real GitHub issues and the pull requests that resolved them, and ProgramBench ([60]) uses existing programs as references and generates behavioral tests by agent-driven fuzzing. Agents increasingly author the questions as well: OfficeQA Pro V2 ([11]) generates them with synthesis, solver, and verification agents and keeps humans only for the final review. JFinBench likewise builds on existing artifacts, namely the official exam PDFs, and makes their conversion scalable with a human-in-the-loop agentic pipeline, in which a coding agent automates most of the work and a human verifies the results.
JFinBench
JFinBench is a benchmark of Japanese financial capabilities comprising 321 examples drawn from six certification exams held between May and August 2026, as summarized in Table 2.
| Exam | Exam date | Sections | MCQ | Short | Form | Essay | Examples | w/ image |
| CPA short-answer exam | May 24, 2026 | 4 | 93 | – | – | – | 93 | – |
| Certified Public Tax Accountant | Aug 4–6, 2026 | 11 | – | – | 17 | 23 | 40 | 1 |
| FP Level 1, written | May 24, 2026 | 2 | 50 | 15 | – | – | 65 | 6 |
| Financial counter services Level 1 | May 2026* | 2 | 50 | 16 | – | – | 66 | 5 |
| Nissho Bookkeeping Level 1 | Jun 14, 2026 | 4 | – | 23 | 1 | – | 24 | – |
| Zensho Bookkeeping Level 1 | Jun 28, 2026 | 2 | – | 33 | – | – | 33 | – |
| Total | 25 | 193 | 87 | 18 | 23 | 321 | 12 |
Table 2. Statistics of JFinBench. Examples is the number of separately graded units, as defined in § 3.3. Question types: MCQ (multiple choice), Short (short answer), Form (structured answer forms), and Essay. The w/ image column counts examples whose question includes an image. *Administered by computer-based testing (CBT) without a fixed exam day. We use the question set published in May 2026.
Source exams
We select exams that cover a core area of Japanese financial practice, whose questions are published as official PDFs by the administering organization with publicly available answer keys, and whose most recent edition was held in 2026. When an exam is offered at multiple levels, we use the highest, Level 1. Below, we give an overview of each of the six exams.1
CPA short-answer exam. The first-stage screening exam of Japan’s national CPA qualification, administered by the Certified Public Accountants and Auditing Oversight Board under the Financial Services Agency. The exam consists of four subjects, namely financial accounting, management accounting, auditing, and corporate law, all answered as multiple-choice questions (MCQs).
Certified Public Tax Accountant exam. The national licensing exam for tax practitioners, administered by the National Tax Council under the National Tax Agency. The exam consists of 11 subjects, two in accounting and nine in tax law, all answered as essays or long-form calculations.
FP skill test. The national certification for financial planners at the practitioner level, consisting of a written test and a practical test. We include the written test. It comprises a basic part, which asks knowledge MCQs on topics such as pensions, insurance, tax planning, and inheritance, and an applied part, which poses client scenarios whose answers, such as pension benefits or inheritance-tax amounts for a given household, must be computed.
Financial counter services skill test. The national certification for bank-counter professionals, combining knowledge MCQs with counter consultations answered as MCQs and short responses. Nissho Bookkeeping. Japan’s de facto standard bookkeeping certification. Level 1 covers advanced financial and cost accounting, such as consolidated statements and standard costing, answered mostly as short computed values, with one full answer form.
Zensho Bookkeeping. A bookkeeping test administered by the National Association of Commercial High Schools and taken mainly by commercial high school students, covering financial and cost accounting answered mostly as short computed values.
Construction pipeline
The exam PDFs differ in layout across exams and often across questions within an exam, and their tables, answer forms, and figures do not survive standard text extraction tools such as PyMuPDF (PyMuPDF, 2015). We therefore build JFinBench with the human-in-the-loop agentic pipeline shown in Figure 1, which converts the PDFs into structured, machine-readable examples in three steps: generic scripts shared by all exams transcribe the pages, a coding agent writes exam-specific code that turns the transcripts into examples, and a human verifies every example and feeds corrections back to the agent. We describe each step below.
Generic transcription. Shared scripts render each page as an image at 300 dpi and transcribe it into markdown with a VLM, Gemini 3.8 Flash [17], which preserves the page layout: tables become markdown tables, and figures that cannot be transcribed as text are replaced by explicit placeholders rather than silently dropped. Each transcript is compared with the PyMuPDF-extracted text as character multisets, flagging omissions and hallucinations for inspection.
Exam-specific conversion by a coding agent. For each exam, a coding agent, Claude Code [2], writes the code that splits the transcripts into typed example records, attaches options and answer keys, crops figure regions for visual examples, removes non-content artifacts such as page headers and a preparatory school’s predicted point allocations, and generates a side-by-side review page, shown in Appendix § 7.§ 3. The agent works from natural-language instructions that specify the source URLs and the statutory basis date, so adding an exam requires no manual programming.
Human verification and feedback. For every example, one of the authors, a native Japanese speaker, compares the final record against the source page on this review page. The reviewer reports problems in natural language, ranging from misparsed options and leftover pagination notes to missing reference material and unclear answer formats, and the agent fixes its code or the records and regenerates the affected examples for re-verification. All 321 examples were verified this way, which took roughly two hours per exam, or about 12 hours in total. The six exams differ widely in format, from multiple choice to blank-filling bookkeeping forms and tax essays across 11 subjects, yet share the same generic scripts, so adding a new exam or edition requires only new agent-written code and human verification. Since neither the generic scripts nor the agent-driven workflow is specific to finance, the same pipeline could be applied to other certification exams, such as the national medical licensing or bar exams.
Statistics and question types
Table 2 breaks the 321 examples down by exam and question type. JFinBench has four question types, each with its own grading method described in § 4.3. MCQ examples are answered by selecting an option. Short answer examples ask for a term or an amount written directly. Form examples require a structured answer form with up to dozens of interdependent blanks, graded blank by blank. Essay examples ask for a long-form theory answer, graded with a rubric derived from the answer key. The 321 examples, each a separately graded unit, derive from 267 source problems. Multiple choice accounts for 193 examples. The remaining 128 require generated answers, from one-line values to full essays. Twelve examples additionally include images, such as land-plot diagrams, served to the models as visual inputs. Representative examples are shown in Appendix § 7.§ 2.
Experimental Setup
We evaluate models primarily without tools, answering each example in a single API call, so that scores reflect the knowledge stored in the model’s parameters and its reasoning. § 5.5 additionally evaluates a setting with tools, where models can use a terminal and web access to run computations and retrieve knowledge they lack.
Models
We evaluate 12 models: Claude Fable 5 [3], GPT-6 Astra (OpenAI, 2026a), and Gemini 3.8 Flash [17] as proprietary models, and Kimi K3.1 Pro, K3, and K2.6 [27, 33 b], DeepSeek V4.1 Flash 1017 [12 b], Qwen3.8 Flash [46], MiMo-V2.6-Pro and MiMo-V2.6-Flash [56], GLM 5.3 Flash [16], and Mistral Large 4 [32] as open-weight models. All models accept image inputs. Gemini 3.8 Flash is accessed through Google’s official API. All other models are accessed through the OpenRouter API,2 pinned to the model developer’s official provider, except for DeepSeek V4.1 Flash 1017, which is served by DeepInfra. Release dates and reported knowledge cutoffs are listed in Appendix § 8.§ 1, and every exam postdates every reported cutoff.
Inference settings
We use the developer’s recommended sampling parameters where published. We set the maximum output length to 131,072 tokens. For each model, we run the maximum and the minimum reasoning effort it supports. Models without an effective effort control use a single setting with thinking enabled. Because neither model outputs nor judge decisions are deterministic, every combination of model and effort level is run three times, and we report the mean.
Scoring
Each example is scored by a method matched to its question type.
MCQ: regex-based answer extraction, with an LLM-judge fallback [62] for malformed outputs.
Short answer: an equivalence judge that accepts mathematically or legally equivalent phrasings.
Form: blank-level partial credit, i.e., the fraction of blanks matching the answer key, with up to about 40 blanks per form.
Essay: rubric-based grading, where the key points of the answer key serve as the rubric and the score is the fraction of key points the response covers.
All judges use the same grader model, GPT-5.1 (OpenAI, 2025). The full judge prompts are in Appendix § 8.§ 4, and the sensitivity of scores to the judge model is analyzed in Appendix § 8.§ 5, where re-grading with a second judge, Claude Opus 5.5 [4], yields closely matching scores. An exam’s score is the mean over its examples, and the overall score is the mean over the six exams with equal weight. Cost is computed from measured input/output tokens at the official list prices given in Appendix § 8.§ 2 and excludes the grader.
Prompts
Every prompt states the exam context together with the exam date and the statutory basis date, for example that “laws in force as of April 1, 2026 apply,” because the correct answer to many questions can change with revisions of statutes, regulations, and accounting standards. Models are thus graded under the law in force at exam time. Answer-format instructions are specific to the question type, and the full MCQ template is in Appendix § 8.§ 3.
Results
Main results
| Model | Score by exam (%) | Overall | |||||||
| CPA | Tax | FP | Counter | Nissho | Zensho | Score (%) | Cost ($/Q) | Tok. (K/Q) | ||
| Proprietary models | |||||||||
| Claude Fable 5 (max) | 97.1 | 79.4 | 100.0 | 100.0 | 100.0 | 98.0 | 95.8±0.4 | 0.555 | 10.9 |
| GPT-6 Astra (max) | 96.8 | 73.7 | 99.5 | 99.0 | 96.9 | 96.0 | 93.6±0.6 | 0.248 | 4.8 |
| Gemini 3.8 Flash (high) | 92.5 | 60.7 | 96.9 | 98.5 | 98.2 | 85.9 | 88.8±1.3 | 0.025 | 6.6 |
| Open-weight models | |||||||||
| Kimi K3.1 Pro (max) | 84.2 | 48.1 | 87.7 | 90.9 | 96.8 | 92.9 | 83.4±0.9 | 0.145 | 14.3 |
| Kimi K3 (max) | 78.9 | 48.3 | 81.0 | 85.9 | 92.6 | 92.9 | 79.9±2.3 | 0.253 | 16.6 |
| DS V4.1 Flash 1017 (max) | 73.1 | 39.9 | 66.2 | 80.8 | 89.9 | 88.9 | 73.1±0.7 | 0.007 | 23.5 |
| Qwen3.8 Flash (thinking) | 65.9 | 28.8 | 39.5 | 73.7 | 87.4 | 79.8 | 62.5±1.0 | 0.020 | 43.2 |
| MiMo-V2.6-Pro (thinking) | 63.4 | 23.1 | 50.3 | 76.3 | 76.4 | 82.8 | 62.1±1.0 | 0.030 | 33.9 |
| Kimi K2.6 (thinking) | 54.5 | 25.2 | 42.1 | 74.8 | 84.6 | 73.7 | 59.1±0.5 | 0.058 | 14.2 |
| Mistral Large 4 (thinking) | 54.8 | 25.5 | 44.1 | 76.3 | 74.8 | 73.7 | 58.2±2.3 | 0.061 | 29.0 |
| GLM 5.3 Flash (max) | 56.6 | 23.6 | 40.0 | 65.7 | 80.4 | 79.8 | 57.7±2.3 | 0.013 | 26.4 |
| MiMo-V2.6-Flash (thinking) | 50.5 | 18.5 | 36.4 | 62.1 | 65.3 | 72.7 | 50.9±1.0 | 0.010 | 34.4 |
Table 3. Model performance on JFinBench (w/o tools) at maximum reasoning effort (mean over 3 runs, with the standard deviation for the overall score). The best open-weight model trails the best proprietary model by 12 points. Full results with standard deviations are in Appendix § 9.
Table 3 reports the main results without tools at maximum reasoning effort. Claude Fable 5 and GPT-6 Astra achieve the highest overall scores of 95.8 and 93.6. The two leaders score 96–100 on every exam except the tax accountant exam, where the best score is 79.4. The three proprietary models occupy the top three ranks. The strongest open-weight model, Kimi K3.1 Pro, reaches an overall score of 83.4, a full 12 percentage points behind Claude Fable 5, and every open-weight model scores below every proprietary one. This gap contrasts with agentic coding, a capability that model developers actively prioritize, where Kimi K3.1 Pro is reported to score 65.7% on Terminal-Bench 4.0, above GPT-6 Astra’s 58.2% [34]. On JFinBench, GPT-6 Astra instead leads Kimi K3.1 Pro by 10.2 points. This clear separation suggests that open-weight models lack much of the specialized knowledge of Japanese finance that proprietary models have acquired.
Performance vs. cost
Reasoning models [18] (OpenAI 2024b) expose settings, such as reasoning effort, that control how much test-time compute they spend, so models should also be compared on the cost at which they reach a given score [7]. Figure 2 therefore plots overall score against API cost per question, at the lowest and the maximum reasoning-effort settings. For every model, raising the effort increases both the score and the cost. Gemini 3.8 Flash stands out for its low cost, outscoring every open-weight model even at low effort. Appendix § 9 additionally reports the full low-effort results and plots score against output tokens.

Figure 2. Model performance vs. cost (w/o tools). For every model with an effort setting, raising the reasoning effort increases both cost and score, showing a trade-off between the two.
Performance vs. parameter count
Model scale is considered an important factor underlying performance on knowledge-intensive benchmarks [1, 15, 48]. Figure 3 plots overall score against total rather than active parameter count for the nine open-weight models, whose parameter counts are public. Score generally increases with parameter count (Pearson ), but parameter count is not the only factor. Kimi K3.1 Pro, post-trained from Kimi K3, improves on it by 3.5 points at the same 2.8T parameters, which points to the effect of post-training, and DeepSeek V4.1 Flash 1017 outscores the trillion-parameter Kimi K2.6, MiMo-V2.6-Pro, and Mistral Large 4 with about half their parameters.

Figure 3. Overall score (w/o tools) versus total parameter count for the open-weight models (maximum effort). Score correlates positively with parameter count (Pearson ).
Failure analysis
To understand why models fail, we classify every failed example (score < 0.5) from the first maximum-effort run of each evaluated model into one of six primary causes.
Knowledge: the response lacks or misstates the fact or rule the question tests.
Misreading / reasoning: the response misreads the question or draws a wrong conclusion from correct knowledge.
Calculation: arithmetic errors alone.
Format / incomplete: the answer deviates from the specified format, is missing required parts,
or is unparseable.
Token-limit truncation: generation, including reasoning tokens, hits the output-token limit with no final answer.
Grading artifact: the response is equivalent to the answer key but was scored as wrong.
Token-limit truncation is detected automatically. The other categories are assigned by an LLM classifier (GPT-5.1), following the LLM-judged failure analysis of Terminal-Bench [31]. The classifier reads the question, the answer key, and the model’s response, and picks the first error that derails the response. To validate the classifier, one of the authors independently labeled 30 sampled failures, five per category, including automatically detected token-limit truncation. The labels agree on 26 of 30 (Cohen’s ), and all four disagreements are between the knowledge and misreading/reasoning categories. Figure 4 shows the distribution.

Figure 4. Error type distribution of each model (w/o tools, failed examples with score < 0.5, run 1 at maximum effort). Knowledge errors are the most common cause for most models, and token-limit truncation is concentrated in a handful of models.
Knowledge errors dominate. Knowledge is the most common primary cause overall. A recurring case is statute recall, where models state the purpose of a provision correctly but cannot reproduce its enumerated requirements.
Overlong reasoning hits the token limit. Token-limit truncation is concentrated in five open-weight models, the MiMo-V2.6 models, Mistral Large 4, Qwen3.8 Flash, and GLM 5.3 Flash. These models sometimes fail to control the length of their reasoning and keep generating until they hit the output-token limit.
Evaluation with tools
Results without tools measure parametric knowledge and in-context reasoning, but real financial work increasingly involves tools ([6]). We therefore also evaluate every model in a setting with tools, where for each question the model can use a terminal and web access over multiple turns before answering. We run this evaluation with Harbor ([51]), a framework for agentic evaluation that runs each task in an isolated container. Each model is evaluated with Harbor’s Terminus-2 agent harness, with mini-SWE-agent ([59]), and with the developer’s official harness where one exists, and we report the score of its best harness. Because the web may contain the published answers, the instructions prohibit consulting answer keys, and we audit every trajectory with an LLM. Attempts that used answer material, 8 of the 11,556 with Terminus-2 (0.07%), score 0 (Appendix § 10.§ 1). Appendix § 10 describes the setup in detail and reports further analyses.
Tools raise every model. Table 4 compares the two settings. Tool access raises the score of every model, and most for GLM 5.3 Flash, which gains 29.9 points. Taking each model’s best harness, the gap between the best open-weight and the best proprietary model narrows from 12.4 to 4.4 points, with Kimi K3.1 Pro reaching 92.9 with Kimi Code against GPT-6 Astra’s 97.3 with Terminus-2. Yet tools do not erase the differences between models. The ranking changes little between the two settings (Spearman = 0.87 over the 12 models), and the same three models lead in both. The main exception is GLM 5.3 Flash, which rises from 11th to 7th. Models that know more without tools thus also score higher with them, suggesting that knowledge stored in the parameters helps an agent decide what to look up and how to apply what it finds. Retrieval complements rather than replaces this knowledge, so JFinBench remains informative when models can use tools. Appendix § 10.§ 3 compares the harnesses in detail.
| Model | w/o tools | w/ tools (harness) | |
| Claude Fable 5 (max) | 95.8 | 96.4 (Terminus-2) | +0.6 |
| GPT-6 Astra (max) | 93.6 | 97.3 (Terminus-2) | +3.7 |
| Gemini 3.8 Flash (high) | 88.8 | 94.2 (Terminus-2) | +5.4 |
| Kimi K3.1 Pro (max) | 83.4 | 92.9 (Kimi Code) | +9.5 |
| Kimi K3 (max) | 79.9 | 91.0 (Kimi Code) | +11.1 |
| DS V4.1 Flash 1017 (max) | 73.1 | 91.5 (mini-SWE-agent) | +18.4 |
| Qwen3.8 Flash (thinking) | 62.5 | 85.9 (mini-SWE-agent) | +23.4 |
| MiMo-V2.6-Pro (thinking) | 62.1 | 86.3 (mini-SWE-agent) | +24.2 |
| Kimi K2.6 (thinking) | 59.1 | 74.7 (mini-SWE-agent) | +15.6 |
| Mistral Large 4 (thinking) | 58.2 | 75.1 (Mistral Vibe) | +16.9 |
| GLM 5.3 Flash (max) | 57.7 | 87.6 (Terminus-2) | +29.9 |
| MiMo-V2.6-Flash (thinking) | 50.9 | 75.8 (MiMo Code) | +24.9 |
Table 4. Comparison of each model’s score without and with tools (mean over 3 runs). For w/ tools, we report the best score among the Terminus-2, mini-SWE-agent, and official harnesses (Table 11). With shell and web access, every model improves, and the open-weight models gain 9.5 to 29.9 points.
| Model | Terminus-2 | mini-SWE-agent | Official harness |
| Harbor 0.24.0 | 2.4.6 | ||
| Claude Fable 5 | (Claude Code 2.1.306) | ||
| GPT-6 Astra | (Codex 0.163.0) | ||
| Gemini 3.8 Flash | (Gemini CLI 0.65.0) | ||
| Kimi K3.1 Pro | (Kimi Code 2.3.0) | ||
| Kimi K3 | (Kimi Code 2.3.0) | ||
| DS V4.1 Flash 1017 | – | ||
| Qwen3.8 Flash | (Qwen Code 0.25.3) | ||
| MiMo-V2.6-Pro | (MiMo Code 0.1.15) | ||
| Kimi K2.6 | (Kimi Code 2.3.0) | ||
| Mistral Large 4 | (Mistral Vibe 3.0.3) | ||
| GLM 5.3 Flash | – | ||
| MiMo-V2.6-Flash | (MiMo Code 0.1.15) |
Table 11. Overall with-tools score under different agent harnesses (mean±std over 3 runs). A dash marks official harnesses not supported by Harbor 0.24.0.
How agents use tools. To see how the models improve with tools, we analyze which kinds of tools the agents use in the first run, where an attempt is one agent session on one question. GPT-6 Luna ((OpenAI, 2026b)) reads all commands of each attempt and labels whether it involves file operation, web retrieval, calculation, or other commands. Figure 5 shows, for two proprietary and two open-weight models with Terminus-2, the one harness common to all models, the share of the 321 tasks in which the agent uses each kind of tool. Every model operates on files in almost every task, which is expected because the final answer must be written to a file. Web retrieval occurs in 56 to 66% of the tasks, as the agents look up knowledge they lack. Calculation occurs in 9 to 28% of the tasks, where the agents compute amounts such as taxes, depreciation, and present values with Python or bc.

Figure 5. Breakdown of the main tools that four representative models used in the with-tools evaluation (Terminus-2, run 1).
Agents consult the statutes. Figure 6 shows the share of tasks in which the agent accesses each of the most frequent domains. All four models most often consult e-Gov, the government’s statute database,3 and the open-weight models also query search engines far more often, for example DuckDuckGo in 47% of the tasks for GLM 5.3 Flash against at most 11% for the proprietary models.

Figure 6. Breakdown of the domains that four representative models accessed most often in the with-tools evaluation (Terminus-2, run 1).
Figure 12 in Appendix § 10.§ 4 shows a trajectory in which the agent retrieves the statutes in force on the exam’s basis date from e-Gov and answers a question it fails without tools. These results suggest that, with tools, models raise their scores mainly by supplementing missing knowledge and, on computation questions, by calculating exactly.
Conclusion
We introduced JFinBench, a Japanese financial benchmark built from the 2026 editions of six certification exams with a human-in-the-loop agentic pipeline that reduces the human effort of future updates. The leading proprietary models are near the ceiling except on tax practice, while the best open-weight model trails the best proprietary model by 12 points. Missing knowledge is the main challenge. Tool use helps substantially, raising every model and narrowing but not closing the gap between the best proprietary and best open-weight models, from 12 to about 4 points, while the ranking of models stays largely the same. We hope JFinBench encourages progress in domains that general benchmarks overlook.
Limitations
Coverage. We include only exams whose questions are published as PDFs by the administering organization. Exams that do not publish past questions as official PDFs, such as the Securities Analyst (CMA) exam,4 are out of scope.
Contamination. Every exam postdates every reported knowledge cutoff listed in Appendix § 8.§ 1, but this temporal separation does not guarantee the absence of contamination. Most vendors do not report a cutoff at all, and reported cutoffs are self-declared, vary in definition across developers, and may differ from the effective cutoff of the training data [9]. Models may also be post-trained on data collected after the stated date. Refreshing the benchmark with future exam editions remains the primary mitigation.
Saturation. The two leading proprietary models already score 96–100 on every exam except the tax accountant exam, so JFinBench offers limited headroom for differentiating them. The benchmark remains well suited to tracking the progress of open-weight models, which still trail by 12 percentage points. For frontier models, promising directions include filtering future editions to the hardest questions, as in Humanity’s Last Exam [44], commissioning new expert-authored questions [31], and building agentic financial benchmarks [6, 61].
Ethical Considerations
The exam material contains no personal data, as persons appearing in problem statements are fictitious. A high JFinBench score does not constitute a professional qualification, and our results should not be read as evidence that current models can substitute for licensed tax accountants or CPAs. Because JFinBench is built from officially published examinations, we see little risk of social harm or misuse arising from the benchmark itself.
Acknowledgments
We thank Alain Saas for helpful discussions.
Benchmark Details
Exam Names, Sources, and Licenses
Table 5 lists the official Japanese name of each exam, the source and license of its questions and answer keys, and how we distribute them. The questions and answer keys come from the publishers’ official pages, except that the answer keys of the Certified Public Tax Accountant exam and Nissho Bookkeeping are the answer bulletins of the preparatory school TAC. Copyright, where applicable, remains with the respective rights holders, and the hosted materials are provided under the source-specific terms in the table. JFinBench was prepared by the authors from the publishers’ PDFs and is not an official publication of, or endorsed by, the examination bodies or answer providers. Our code is released under the Apache License 2.0.
| Exam | Questions Source | Questions License | Answer keys Source | Answer keys License | Distribution |
| CPA short-answer exam (公認会計士短答式試験) | CPA and Auditing Oversight Board (FSA), question PDFs | Public Data License 1.0 (CC BY 4.0 compatible) | CPA and Auditing Oversight Board (FSA), official answer PDF | Public Data License 1.0 | Hugging Face |
| Certified Public Tax Accountant exam (税理士試験) | National Tax Agency, question and answer-form PDFs | Public Data License 1.0 (CC BY 4.0 compatible) | TAC Co., Ltd., answer bulletins | All rights reserved | Hugging Face (questions), recipe (answer keys) |
| FP skill test (FP技能検定1級学科) | Kinzai, question PDFs | Kinzai’s exam-question usage terms (attribution and modification notice, no application required) | Kinzai, official answer PDF | Kinzai’s usage terms cover the questions; coverage of the model answers is being confirmed | Hugging Face (questions; the basic section’s answers are option numbers), recipe (model answers of the applied section) |
| Financial counter services skill test (金融窓口サービス技能検定1級) | Kinzai, question PDFs | Kinzai’s exam-question usage terms (as above) | Kinzai, question PDFs (the model answers follow each question) | Kinzai’s usage terms cover the questions; the hosted answers are option symbols and figures | Hugging Face |
| Nissho Bookkeeping (日商簿記検定1級) | Japan Chamber of Commerce and Industry, question PDFs | All rights reserved | TAC Co., Ltd., answer bulletins | All rights reserved | Recipe |
| Zensho Bookkeeping (全商簿記実務検定1級) | National Association of Commercial High Schools, question PDFs | All rights reserved, prior permission required beyond educational use | National Association of Commercial High Schools, official answer PDF | All rights reserved, prior permission required beyond educational use | Recipe |
Table 5. Official Japanese name of each exam, the source and license of its questions and answer keys, and how we distribute them: on Hugging Face, or as recipes that rebuild the examples from the publishers’ PDFs (Distribution, below). Entries link to the source pages or terms.
Distribution. “Hugging Face” in Table 5 means that the materials are included in the hosted dataset. “Recipe” means that they are not redistributed: users obtain the PDFs from the publishers and rebuild the materials locally with recipes provided in the repository (copy operations into the text layer of the PDFs plus our own formatting, containing no exam text, verified by per-example hashes). The answer field of such items is null in the hosted dataset. A recipe grants no permission to use the source materials; users need their own legal basis, such as a licence or a statutory exception.
Representative Examples
Figure 7 shows an MCQ example and a VQA example of JFinBench.

Figure 7. Two representative examples of JFinBench: a statutory-knowledge MCQ and an image-dependent long-form tax computation (abridged). Sources: the CPA short-answer exam (Certified Public Accountants and Auditing Oversight Board) and the Certified Public Tax Accountant exam (National Tax Agency), both under the Public Data License 1.0, transcribed and restructured by the authors.
Human Verification Interface
Figure 8 shows the review interface used in the final step of the pipeline described in § 3.2.

Figure 8. Human verification interface (shown: CPA short-answer exam, auditing). Left: the source exam PDF page. Right: the final record with example ID (cpa-tanto2-r08-kansa-q01), question, options, and the answer key with the corresponding region of the answer-key PDF.
Representative errors that verification caught in early build passes, and that motivate keeping a human in the loop:
List/option confusion: enumerated conditions inside a problem statement misparsed as the answer options of an MCQ example.
Pagination artifacts: page-boundary text such as 「(次ページへ続く)」 (continued on the next page) and repeated 「(続き)」 (continued) sub-headings leaking into question text when a problem spans pages.
| Model | Released | Knowledge cutoff |
| GPT-6 Astra | Sep 2026 | Apr 30, 2026 |
| Claude Fable 5 | Jun 2026 | Jan 2026 |
| Gemini 3.8 Flash | Sep 2026 | Mar 2026 |
| Kimi K3.1 Pro | Oct 2026 | Not reported |
| Kimi K3 | Jul 2026 | Not reported |
| Kimi K2.6 | Apr 2026 | Not reported |
| GLM 5.3 Flash | Aug 2026 | Not reported |
| Qwen3.8 Flash | Aug 2026 | Not reported |
| DS V4.1 Flash 1017 | Oct 2026 | Not reported |
| MiMo-V2.6-Pro | Sep 2026 | Not reported |
| MiMo-V2.6-Flash | Sep 2026 | Not reported |
| Mistral Large 4 | Oct 2026 | Not reported |
Table 6. Release dates and developer-reported knowledge cutoffs.
Missing or improper figure crops: image-dependent examples initially built as text-only despite a figure in the source, and crop bounding boxes that cut off part of a diagram.
Evaluation Setup Details
Model Release Dates and Knowledge Cutoffs
Table 6 lists, for each evaluated model, its public release date and its knowledge cutoff as reported by the developer.
Model Prices
Table Table 7 lists the list prices used to compute the costs reported in this paper, retrieved between September 30 and October 24, 2026 from each model’s serving endpoint described in § 4.
| Model | Input | Output |
| GPT-6 Astra | 10.00 | 50.00 |
| Claude Fable 5 | 10.00 | 50.00 |
| Gemini 3.8 Flash | 0.75 | 3.75 |
| Kimi K3.1 Pro | 2.00 | 10.00 |
| Kimi K3 | 3.00 | 15.00 |
| Kimi K2.6 | 0.95 | 4.00 |
| GLM 5.3 Flash | 0.15 | 0.50 |
| Qwen3.8 Flash | 0.15 | 0.47 |
| DS V4.1 Flash 1017 | 0.098 | 0.294 |
| MiMo-V2.6-Pro | 0.43 | 0.87 |
| MiMo-V2.6-Flash | 0.14 | 0.28 |
| Mistral Large 4 | 0.68 | 2.09 |
| GPT-5.1 (grader) | 1.25 | 10.00 |
Table 7. List prices (USD per 1M tokens) used for cost computation.
Prompt Example
The MCQ prompt template for the CPA short-answer exam is shown below, with an English translation below the dashed line.
| PROMPT | MCQ template (CPA short-answer exam) |
| 以下は、2026年5月24日に実施された令和8年公認会計士試験第II回短答式試験({subject})の問題です。解答に当たり適用すべき法令基準等は、2026年4月1日現在施行(適用)のものとします。問題に記載された条件・指示に従って解答してください。 | |
| {question} | |
| {options} | |
| ステップバイステップで考えてから答えてください。最後の行は ‘Answer: $NUMBER’ の形式($NUMBERは選択肢の番号)で回答してください。 |
Table 12.
The following is a question from the Reiwa-8 Certified Public Accountants Examination, second short-answer test ({subject}), held on May 24, 2026. The laws and standards to apply in your answers are those in force (applicable) as of April 1, 2026. Answer according to the conditions and instructions stated in the question.
{question}
{options}
Think step by step before answering. End your response with a final line in the form ‘Answer: $NUMBER’, where $NUMBER is the number of the chosen option.
Grading Prompts
Placeholders [question], [correct_answer], and [response] are substituted per example. Each box shows the Japanese prompt that the judge receives, with an English translation below the dashed line.
The equivalence judge is used for short free-response examples with binary scoring.
PROMPT Equivalence judge
あなたは日本の金融・会計系資格試験の採点者です。[question]に対する[response]の最終解答が、模範解答[correct_answer]と一致するか判定してください。 判定基準:
[correct_answer]に複数の空欄(①②③...)がある場合、すべての空欄が一致して初めて正解とする。1つでも不一致があれば不正解。
数値は表記が異なっても、単位を換算して同値であれば一致とみなす。例:「1年6カ月」と「18カ月」、「1億6,000万円」と「16,000万円」、「3%」と「3%」、「1,200万円」と「1200万円」は同値。
漢数字と算用数字、全角と半角、桁区切りカンマの有無、括弧内の単位表記の有無は同値として扱う。
語句の空欄は、模範解答と同一の概念・制度を指す一般に認められた表現であれば一致とみなす。
計算過程の正誤やスタイルは採点対象外。最終的な解答の値のみで判定する。
[response]から最終解答が特定できない場合は不正解とする。
出力形式: extracted_final_answer / reasoning / correct (yes/no) / confidence
You are a grader for Japanese financial and accounting certification exams. Judge whether the final answer in [response] to [question] matches the model answer [correct_answer]. Criteria:
If [correct_answer] contains multiple blanks (123...), the answer is correct only if every blank matches. A single mismatch makes it incorrect.
Numbers written differently count as matching if they are equivalent after unit conversion, e.g., “1 year and 6 months” and “18 months”, “160 million yen” and “1.6 hundred-million yen”.
Kanji versus Arabic numerals, full-width versus half-width characters, digit-separator commas, and parenthesized units are treated as equivalent.
For term blanks, any generally accepted expression denoting the same concept or institution as the model answer counts as a match.
The correctness or style of intermediate calculations is not graded. Judge only the final answer value.
If no final answer can be identified in [response], it is incorrect.
Output format: extracted_final_answer / reasoning / correct (yes/no) / confidence
The blank-level partial-credit judge is used for large answer forms (score = matched blanks / total blanks).
PROMPT Blank-level partial-credit judge
あなたは日本の金融・会計系資格試験の採点者です。[question]に対する[response]を、模範解答[correct_answer]と空欄(記入箇所)単位で照合してください。 判定基準:
[correct_answer]の各記入箇所(表の空欄・金額・科目名など)について、[response]の対応する箇所が一致しているかを1つずつ判定する。
数値は表記が異なっても単位換算して同値なら一致(桁区切り・全角半角の差も同値)。
勘定科目名は同一概念を指す一般に認められた表現なら一致。
[response]に対応する記入がない箇所は不一致として数える。
模範解答で0や記入なしとされる箇所も1箇所として数える。
出力形式: reasoning / matched_blanks(整数)/ total_blanks(整数)
You are a grader for Japanese financial and accounting certification exams. Compare [response] to [question] against the model answer [correct_answer] blank by blank. Criteria:
For each entry in [correct_answer] (table blanks, amounts, account titles, etc.), judge one by one whether the corresponding entry in [response] matches.
Numbers match if equivalent after unit conversion (digit-separator and width differences are also equivalent).
Account titles match if they are generally accepted expressions for the same concept.
Entries with no corresponding answer in [response] count as mismatches.
Entries that the model answer leaves as 0 or blank also count as one entry.
Output format: reasoning / matched_blanks (integer) / total_blanks (integer)
The key-point coverage judge is used for tax-law theory and essays (score = covered points / total points).
PROMPT Key-point coverage judge
あなたは日本の税務・会計系資格試験の理論・論述問題の採点者です。模範解答[correct_answer]を採点基準として、[question]に対する受験者の解答[response]を採点してください。 手順:
まず[correct_answer]から採点上の要点を列挙する。要点とは次のいずれかに当たる独立した内容単位である:結論(金額・判定・可否・対象者など)/根拠となる規定・制度の内容(成立要件の各号、原則と例外、範囲・限度など)/設例の事実への当てはめ。見出し・配点記号・繰り返しは要点に数えない。
[response]が各要点を実質的にカバーしているかを1つずつ判定する。表現の違い・要約・条文番号の有無は許容する(内容が正しく伝わればカバー)。結論の金額・判定が模範解答と異なる場合、その要点はカバーとみなさない。模範解答にない誤った記述があっても減点はしないが、要点と矛盾する記述はその要点を不カバーとする。
出力形式: reasoning / covered_points(整数)/ total_points(整数)
You are a grader for theory and essay questions in Japanese tax and accounting certification exams. Using the model answer [correct_answer] as the grading key, grade the examinee’s answer [response] to [question]. Procedure:
First, enumerate the key points of [correct_answer]. A key point is an independent unit of content that is one of the following: a conclusion (amount, determination, eligibility, affected parties, etc.); the content of the underlying rule or institution (individual requirements, principle and exceptions, scope or limits, etc.); or the application to the facts of the case. Headings, point-allocation marks,
and repetitions do not count as key points.
Judge one by one whether [response] substantively covers each key point. Differences in wording, summarization, and the presence or absence of article numbers are acceptable (a point is covered if its content is conveyed correctly). If a concluding amount or determination differs from the model answer, that point is not covered. Incorrect statements absent from the model answer are not penalized, but a statement contradicting a key point makes that point uncovered.
Output format: reasoning / covered_points (integer) / total_points (integer)
Judge Agreement
We re-graded all 128 judge-scored examples of each model (run 1 at maximum effort) with a second judge, Claude Opus 5.5 ([4]), using the same prompts. As Table 8 shows, the two judges agree closely. Pooled over all regrades, their scores correlate at Pearson , and substituting the second judge’s scores changes each model’s overall score by at most 1.7 points without changing the ranking.
| Model | Identical | Overall (run 1) | ||
| GPT-5.1 | Opus 5.5 | |||
| Claude Fable 5 | 0.92 | 103/128 | 95.3 | 95.1 |
| GPT-6 Astra | 0.72 | 94/128 | 93.8 | 94.6 |
| Gemini 3.8 Flash | 0.89 | 95/126 | 88.1 | 88.2 |
| Kimi K3.1 Pro | 0.79 | 89/125 | 83.1 | 82.5 |
| Kimi K3 | 0.94 | 96/128 | 77.4 | 77.2 |
| DS V4.1 Flash 1017 | 0.97 | 97/126 | 72.4 | 73.3 |
| Qwen3.8 Flash | 0.95 | 103/128 | 62.9 | 62.0 |
| MiMo-V2.6-Pro | 0.94 | 105/128 | 61.9 | 60.5 |
| Kimi K2.6 | 0.95 | 93/128 | 59.3 | 59.2 |
| Mistral Large 4 | 0.95 | 94/127 | 60.2 | 59.4 |
| GLM 5.3 Flash | 0.99 | 100/128 | 58.0 | 58.5 |
| MiMo-V2.6-Flash | 0.94 | 107/128 | 51.8 | 50.1 |
Table 8. Agreement between the two judge models over the 128 judge-scored examples per model (w/o tools) (125 for Kimi K3.1 Pro, 126 for Gemini 3.8 Flash and DeepSeek V4.1 Flash 1017, and 127 for Mistral Large 4, after excluding eight failed second-judge calls). is the example-level Pearson correlation, Identical is the number of examples with identical scores, and the last two columns are the overall score (run 1) under each judge.
Additional Results without Tools
Table 9 reports the full results at both the maximum and the lowest reasoning-effort settings, with standard deviations for every column. Figure 9 plots overall score against average output tokens per question at both effort settings, and Figure 10 and Figure 11 break the cost and token analyses down by exam.
| Model | CPA | Tax | FP | Counter | Nissho | Zensho | Score (%) | Cost ($/Q) | Tok. (K/Q) | |
| Proprietary models | |||||||||
| Claude Fable 5 (max) | 97.10.6 | 79.42.4 | 100.00.0 | 100.00.0 | 100.00.1 | 98.01.7 | 95.80.4 | 0.5550.010 | 10.90.2 |
| Claude Fable 5 (low) | 95.31.6 | 71.43.0 | 97.92.4 | 98.00.9 | 96.92.4 | 96.01.7 | 92.61.4 | 0.0600.001 | 0.90.0 |
| GPT-6 Astra (max) | 96.80.0 | 73.71.9 | 99.50.9 | 99.00.9 | 96.92.5 | 96.01.7 | 93.60.6 | 0.2480.003 | 4.80.1 |
| GPT-6 Astra (low) | 96.80.0 | 72.01.0 | 96.90.0 | 100.00.0 | 92.82.4 | 93.93.0 | 92.10.1 | 0.0450.000 | 0.70.0 |
| Gemini 3.8 Flash (high) | 92.51.9 | 60.71.4 | 96.91.5 | 98.51.5 | 98.22.4 | 85.93.5 | 88.81.3 | 0.0250.000 | 6.60.0 |
| Gemini 3.8 Flash (low) | 91.80.6 | 51.80.5 | 89.72.4 | 96.50.9 | 96.34.1 | 84.83.0 | 85.10.3 | 0.0050.000 | 1.10.0 |
| Open-weight models | |||||||||
| Kimi K3.1 Pro (max) | 84.22.5 | 48.10.9 | 87.71.5 | 90.91.5 | 96.82.9 | 92.91.7 | 83.40.9 | 0.1450.001 | 14.30.1 |
| Kimi K3.1 Pro (low) | 76.74.3 | 45.43.9 | 81.04.4 | 90.94.0 | 91.50.1 | 84.88.0 | 78.41.7 | 0.0420.001 | 3.90.1 |
| Kimi K3 (max) | 78.95.0 | 48.32.7 | 81.05.8 | 85.92.3 | 92.62.7 | 92.91.7 | 79.92.3 | 0.2530.007 | 16.60.4 |
| Kimi K3 (low) | 67.03.8 | 37.93.5 | 71.82.4 | 86.93.2 | 82.31.1 | 88.93.5 | 72.51.9 | 0.0310.002 | 1.80.2 |
| DS V4.1 Flash 1017 (max) | 73.11.9 | 39.92.9 | 66.21.5 | 80.82.3 | 89.92.1 | 88.91.7 | 73.10.7 | 0.0070.000 | 23.50.3 |
| DS V4.1 Flash 1017 (low) | 72.03.2 | 39.11.3 | 67.71.5 | 82.34.6 | 84.54.6 | 84.83.0 | 71.71.5 | 0.0030.000 | 9.00.0 |
| Qwen3.8 Flash (thinking) | 65.90.6 | 28.82.1 | 39.56.4 | 73.72.3 | 87.40.1 | 79.83.5 | 62.51.0 | 0.0200.000 | 43.20.8 |
| MiMo-V2.6-Pro (thinking) | 63.44.9 | 23.10.4 | 50.36.4 | 76.36.3 | 76.44.8 | 82.81.7 | 62.11.0 | 0.0300.004 | 33.95.0 |
| Kimi K2.6 (thinking) | 54.50.6 | 25.21.3 | 42.14.7 | 74.89.3 | 84.62.7 | 73.71.7 | 59.10.5 | 0.0580.004 | 14.21.1 |
| Mistral Large 4 (thinking) | 54.83.7 | 25.54.7 | 44.13.9 | 76.32.3 | 74.82.3 | 73.74.6 | 58.22.3 | 0.0610.006 | 29.03.0 |
| GLM 5.3 Flash (max) | 56.65.5 | 23.61.9 | 40.01.5 | 65.72.3 | 80.48.6 | 79.81.7 | 57.72.3 | 0.0130.000 | 26.40.5 |
| GLM 5.3 Flash (low) | 45.91.6 | 25.63.8 | 41.03.9 | 62.14.0 | 66.310.8 | 63.63.0 | 50.81.4 | 0.0020.000 | 3.30.6 |
| MiMo-V2.6-Flash (thinking) | 50.55.7 | 18.50.2 | 36.42.4 | 62.11.5 | 65.36.4 | 72.73.0 | 50.91.0 | 0.0100.001 | 34.42.1 |
Table 9. Full results without tools at the maximum and lowest reasoning-effort settings (mean standard deviation over 3 runs). Models without an effective effort control have a single setting with thinking enabled. Bold marks the best score in each score column.

Figure 9. Overall score (w/o tools) versus average output tokens per question (reasoning + answer). Open-weight models tend to generate far longer outputs.

Figure 10. Score (w/o tools) versus cost per question, by exam.

Figure 11. Score (w/o tools) versus output tokens per question, by exam.
Evaluation with Tools
Setup
Inference settings. Each model’s reasoning-effort level, provider pin, and sampling parameters match the maximum-effort runs without tools, except that Gemini is reached through OpenRouter. Each attempt has a wall-clock budget of six hours.
Figure input. Terminus-2 passes only text to the model, so for the figure-dependent examples we add a view_image command to the container. When the agent runs it on an image file, the harness attaches the image to the next observation, so the agent can view any image, as the official harnesses do with their own file-reading tools. The figure is placed in a directory the agent can access, and the instruction states where it is and how to view it.
Detecting answer-key use. In addition to prohibiting answer keys and prep-school solutions in the task instructions, we audit every trajectory with GPT-6 Luna (OpenAI, 2026b), covering all 11,556 Terminus-2 attempts and the 24,196 attempts of the harness comparison. Luna receives a condensed trajectory, consisting of each command (first 2,000 characters) and each tool output (first 1,500 and last 500 characters), and is asked whether the agent opened this exam’s answers, answer bulletins, model answers, or past-exam explanations, whether such an answer appeared in a search snippet that the final answer follows, and whether the agent tampered with grading. Attempts that opened such material, or whose final answer matches an answer shown in a search-result snippet, score 0, regardless of the agent’s intent. Table 10 shows their number per model, 8 in total (0.07%). In each case, the answer was visible in a search-result snippet and the final answer matches it. None of the queries targeted answer keys. They consisted of legal or accounting terms, sometimes with the wording of an option, and several agents explicitly excluded answer pages from their searches. We therefore find no evidence of deliberate answer seeking, but score these attempts 0 because the answer reached the agent.
| Model | Attempts scored 0 |
| Claude Fable 5 | 0 |
| GPT-6 Astra | 0 |
| Gemini 3.8 Flash | 0 |
| Kimi K3.1 Pro | 0 |
| Kimi K3 | 0 |
| DS V4.1 Flash 1017 | 4 |
| Qwen3.8 Flash | 0 |
| MiMo-V2.6-Pro | 0 |
| Kimi K2.6 | 0 |
| Mistral Large 4 | 0 |
| GLM 5.3 Flash | 3 |
| MiMo-V2.6-Flash | 1 |
Table 10. Number of attempts scored 0 for answer-key exposure (with tools, Terminus-2, summed over three runs of 321 tasks).
Intriguing Agent Behavior in Early Experiments
The following behavior was observed in early experiments that we ran before fixing the final evaluation setup, and it is not part of the reported results. Through our oversight, these experiments had three flaws in the with-tools setup: (1) an OpenRouter API key that the agent did not need was placed in its working environment, (2) the image file that the item referred to was missing from that environment, and (3) the agent had no tool for viewing images. The cases below arose while agents were solving an item from the inheritance-tax section of the tax accountant exam. This item should have come with one image containing three figures: a family tree showing the relationships among the heirs, and diagrams of two land plots giving their shapes, dimensions, and road prices. Without them, neither the heirs nor the land values can be determined. On this item, the following two models found the key in their environment and realized that they could use it to get help from other LLMs and APIs.
DeepSeek V4.1 Flash 1017. While looking for a way to read the figures, DeepSeek V4.1 Flash 1017 found the key and made eight API calls to Gemini 2.5 Flash and Pro, having them transcribe as text the formulas that the National Tax Agency publishes only as images and the pages of the question PDF, including the land diagrams, that it downloaded itself. It solved the problem itself from these transcriptions.
TRAJECTORY DeepSeek V4.1 Flash 1017 (mini-SWE-agent, 117 steps, abridged)
Step 3 Finds no figure and searches the files and environment variables for clues.
$ find /workspace /tmp /root -maxdepth 3 -type f | head -100; env | sort | head -100
Step 64 The size-correction formula needed to value a land plot (property valuation guideline 20-2) is published by the National Tax Agency only as an image, so it downloads the image and tries to read it as ASCII art.
$ curl -L -A ’Mozilla/5.0’ -sS
’https://www.nta.go.jp/law/tsutatsu/kihon/sisan/hyoka_new/02/48.gif’ -o /tmp/48.gif;
convert /tmp/48.gif -resize 400% ...
Step 65 Notices the API key in the environment and has Gemini 2.5 Flash read that image through OpenRouter. “. . . Let’s use OpenRouter multimodal API with base64 image to extract text. . . . The environment has API key. Let’s try.”
$ python3 - <<’PY’
img=base64.b64encode(open(’/tmp/48.gif’,’rb’).read()).decode()
payload={"model":"google/gemini-2.5-flash",
"messages":[{"role":"user","content":[{"type":"text","text":
"Extract the mathematical formula from this image exactly. ..."},
{"type":"image_url", ...}]}]}
req=urllib.request.Request(’https://openrouter.ai/api/v1/chat/completions’,
data=json.dumps(payload).encode(),
headers={’Authorization’:’Bearer ’+os.environ[’OPENROUTER_API_KEY’], ...})
Step 66 Repeats the call with Gemini 2.5 Pro for a more accurate reading.
payload={"model":"google/gemini-2.5-pro", ..."Transcribe this Japanese tax formula
image exactly, preserving fractions and symbols. ..."}
Step 76 Looks for the question PDF on the National Tax Agency’s website, judging that this is not cheating because it will extract only the figures. “... retrieving the official question PDF is the only way to solve. I’ll do it, but only extract the figures, not the answer.”
Step 78 Downloads the question PDF.
$ curl -L -A ’Mozilla/5.0’ -sS ’https://www.nta.go.jp/taxes/zeirishi/zeirishishiken/
shikenkekka/76/pdf/mondai_souzoku.pdf’ -o /workspace/mondai_souzoku.pdf
Step 85 Sends page images rendered from the PDF to Gemini 2.5 Pro to read the figures. “Use Gemini vision to extract the figures precisely.”
prompt="""This is a Japanese tax exam page with a land diagram. Transcribe ALL text
and numbers in the diagram, ..."""
Step 86 The call times out on the large images, so it shrinks them and resends.
$ convert /workspace/page7.png -resize 1200x /workspace/page7_small.png; ...
Gemini 3.8 Flash. After realizing that it could not read the figure, Gemini 3.8 Flash found the key, confirmed with a short message that it worked, and then delegated the problem solving itself. It sent the problem text to Gemini 2.5 Flash, asked other models for legal judgments, checks of its own calculations, and details of the figures, and even asked which past or mock exam the item was based on. Partly because it kept asking after receiving inconsistent answers, it made 59 API calls in total, including 33 to Gemini 2.5 Flash and 10 to Perplexity Sonar Pro.
TRAJECTORY Gemini 3.8 Flash (mini-SWE-agent, 261 steps, abridged)
Step 40 Writes the key it found into its code and checks it.
$ python3 -c "import urllib.request, json; api_key = ’[REDACTED]’; print(’API key exists:’, bool(api_key))"
Step 54 Sends a short message to Gemini 2.5 Flash through the OpenRouter API to test the key.
data = json.dumps({’model’: ’google/gemini-2.5-flash’, ’messages’: [{’role’: ’user’, ’content’: ’Hello, are you there?’}]})
req = urllib.request.Request(’https://openrouter.ai/api/v1/chat/completions’, data=data, ...)
Step 66 Sends the problem text to Gemini 2.5 Flash to solve.
prompt = ”’You are an expert in Japanese tax accountant examination (税理士試験相続税法). Please look at this problem description: 令和8年度(第76回)税理士試験相続税法第二問 ...”’
Step 69 Asks Gemini 2.5 Flash which past or mock exam the item is based on.
# Let’s search OpenRouter to ask which past exam or mock exam this is modeled after!
prompt = ”’... the years might have been updated from an earlier past exam like 第73回, 第71回, 第68回, etc., or it is a mock exam from TAC/O-Hara. ...”’
Step 88 Sends dates and amounts from the problem to the web-searching Perplexity Sonar to find the source.
’model’: ’perplexity/sonar’, ’content’: ’In tax accountant exam for Japanese inheritance tax: "令和4年(2022年)5月6日" "1,602,100" ...’
Step 134 Has GPT-4o describe the family-tree figure cropped from the question PDF.
’model’: ’openai/gpt-4o’, ...’Describe everything in this image in extreme detail. It is a family tree diagram ...’
Step 147 Shows Gemini 2.5 Pro a cropped land figure and asks whether the plot is a corner lot.
’Look at Diagram 3 (crop_diagram3.png) very carefully: 1. Do the Top Road and Right Road intersect at the top-right corner to form a corner (角地)? ...’
Step 221 Asks Gemini 2.5 Flash how prep-school mock exams and official solutions round an intermediate result.
’... In TAC / Ohara mock exams and official solutions: How is the applied area of 貸付事業用宅地等 written in the 計算過程? ...’
Step 257 Asks Gemini 2.5 Flash to check its own spousal tax credit calculation.
’Look at the calculation of 配偶者の税額軽減 (Spousal Tax Credit, 相法19条の2): 相続税の総額: 35,595,850円 ...’
Discussion. When we ran these experiments, calling other LLMs was not explicitly prohibited. These actions are therefore strategies that the models devised on their own to complete the task when the image was unavailable, and they may not amount to cheating. Still, these cases are an intriguing illustration of how readily LLM agents exploit resources found in their environment to accomplish a given task. PostTrainBench [47] likewise reports agents using API keys they found to generate synthetic data without authorization. These cases remind us that evaluations giving models a shell and internet access need to be designed for such unexpected behavior, for example by keeping unnecessary API keys out of the agent’s environment.
Effect of the Agent Harness
Table 11 shows each model’s with-tools score with Terminus-2, mini-SWE-agent [59], and the developer’s official harness.
Qualitative Example
Figure 12 shows a representative case. Without tools, Claude Fable 5 fails this CPA question in all three runs because it judges two statements by rules that a March 2026 amendment changed. With tools, it retrieves the statutes in force on the exam’s basis date and answers correctly in all three runs.

Figure 12. Condensed trajectory of Claude Fable 5 solving a CPA financial accounting question with tools (Terminus-2, translated and abridged). Without tools, the model fails this question in all three runs. With tools, it retrieves the statutes in force on the exam’s basis date from the government’s e-Gov statute database and answers correctly.
Score vs. Cost and Output Tokens
Figure 13 and Figure 14 plot scores with tools against cost and output tokens per question. Because each turn re-sends the growing context, costs include provider prompt-cache discounts.

Figure 13. Score (w/ tools, Terminus-2) vs. API cost per question at maximum effort.

Figure 14. Score (w/ tools, Terminus-2) vs. average output tokens per question at maximum effort.
Use of AI Assistants
We used AI systems in the following parts of this work.
Benchmark construction: a coding agent, Claude Code [2], wrote the code that converts the exam PDFs into examples, including the VLM-based transcription and the review pages described in § 3.2. The authors verified every resulting example.
Evaluation and analysis: the evaluated models are LLMs, an LLM judge grades the non-MCQ answers with the prompts in Appendix § 8.§ 4, a second LLM judge checks the grading in Appendix § 8.§ 5, an LLM classifier assigns the failure causes in § 5.4, and an LLM audits the with-tools trajectories for answer-key lookup and classifies their tool use in Appendix § 10.§ 1 and § 5.5.
Paper writing: an AI assistant was used to revise the text, fix typos, and create the charts.
All analyses, claims, and text were reviewed and verified by the authors, who take full responsibility for the content.
Task & Setup
Task (CPA financial accounting, abridged)
Laws as in force on April 1, 2026. Choose the correct pair of statements about Japan’s corporate accounting system.
(a) Under the Companies Act, consolidated accounts may follow U.S. GAAP, designated IFRS, or modified IFRS besides Japanese GAAP.
(b) Temporary financial statements are allowed, but profit up to the temporary closing date may not enter the distributable amount.
(c) Type-2 interim consolidated statements must include the statement of changes in equity and the cash flow statement.
(d) Companies adopting designated or modified IFRS must include the equivalent of consolidated supplementary schedules.
1 (a)(b) 2 (a)(c) 3 (a)(d) 4 (b)(c)
5 (b)(d) 6 (c)(d)
Setting
Terminus-2 drives a shell in a Linux container with internet access and a six-hour budget. The answer goes to /workspace/answer.txt and every consulted URL to sources.txt.
Answer keys and prep-school solutions are prohibited.
Without tools (3 runs)
Answer 3 ((a)(d)) twice and no answer once. The model judges (c) false and (d) true from memory, citing Articles 93 and 94 of the consolidated rules, their numbers before a March 2026 amendment.
Agent Trajectory
Agent Turn 1
Plan: check each statement against the statutes (company accounting rules, Companies Act, interim and consolidated financial statement rules) via the e-Gov statute API.
$ curl -s
"https://elaws.e-gov.go.jp/
api/1/lawdata/411M50000040024"
-o chukanrenketsu.xml; wc -c
chukanrenketsu.xml
Environment
0 chukanrenketsu.xml
// same for all four statutes
Agent Turn 2
All responses are empty. Inspect the HTTP exchange and try the newer API.
$ curl -sv ".../api/1/lawdata/
417AC0000000086" -o /dev/null
Environment
< HTTP/2 301
< date: Mon, 19 Oct 2026
06:55:13 GMT
< location:
https://laws.e-gov.go.jp/
api/1/...
HTTP:200 size:1180738 // API v2
Agent Turns 3–4
The server date is October 2026, so request each statute as in force on the exam’s basis date.
$ curl -sG
".../api/2/law_data/$id"
--data-urlencode
"asof=2026-04-01"
Environment
411M50000040024_20240401_...
// interim consolidated rules, April 2024
version
351M50000040028_20260331_...
// consolidated rules, March 2026
version
Agent Turns 5–6
Parse the downloaded statute XML.
$ python3 show_art.py
keisan.xml ...
Environment
ParseError: syntax error:
line 1, column 0
PExhdyBFcmE9IkhlaXNlaSig...
// the full text is base64-encoded XML
Agent Turns 7–8
After decoding, Articles 120 to 120-3 confirm (a) and Companies Act Article 461(2) refutes (b). Search the interim rules for the type-2 term.
$ python3 - <<'EOF' ... if
term in txt: ... EOF
Environment
// no output: the term does not occur in
the interim rules (2024 version)
Agent Turns 9–10
Search the consolidated rules (2026 version) for the term instead.
$ python3 - <<'EOF'
root=ET.parse('renketsu.xml')
... EOF
Environment
('Art. 190', '(General
principles for type-2 interim
consolidated statements)')
... total: 44
// translated
Agent Turns 11–13
The interim rules were merged into the consolidated rules. Print Article 1, which lists the required statements.
$ python3 show_art.py
renketsu.xml ...
Environment
(i) Consolidated FS (... cash
flow statement and
supplementary schedules, or
under designated IFRS (Art.
312) or modified IFRS (Art.
314) the equivalents of ...
the cash flow statement)
(ii) Type-2 interim
consolidated FS (...
statement of changes in equity
and cash flow statement)
// translated
Agent Turns 14–17
Item (iii) includes the equity statement, so (c) is true. Item (i) asks IFRS adopters for no supplementary schedules, so (d) is false. Write the answer.
$ cat > /workspace/answer.txt
<<'EOF' ... EOF
Finish Answer: 2 ✓ correct
17 turns in 39 minutes | correct in all three with-tools runs
Key Evidence
Why the answer is 2
The March 2026 amendment merged the interim consolidated rules into the consolidated rules and renumbered them. Article 1(1)(iii) of the amended rules lists the statement of changes in equity for type-2 interim statements ((c) true). Article 1(1)(i) asks IFRS adopters only for the equivalents of the primary statements, without supplementary schedules ((d) false).
(a) is true by Articles 120 to 120-3 of the Ordinance on Company Accounting, and (b) is false by Article 461(2) of the Companies Act.
References
- [1]Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. In ICLR, 2025. URL https://openreview.net/forum?id=FXNNiUgtfa.
- [2]Anthropic. Claude Code. https://www.anthropic.com/claude-code, 2025. Accessed 2026-10-13.
- [3]Anthropic. Claude Fable 5 and Claude Mythos 5. https://www.anthropic.com/news/claude-fable-5-mythos-5, 2026a. Accessed 2026-10-04.
- [4]Anthropic. Claude Opus 5.5. https://www.anthropic.com/claude-opus-5-5, 2026b. Accessed 2026-10-04.
- [5]Axel Backlund and Lukas Pettersson. Vending-Bench: A benchmark for long-term coherence of autonomous agents. arXiv preprint arXiv:2502.15840, 2025.arxiv.org/abs/2502.15840
- [6]Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. Finance Agent Benchmark: Benchmarking LLMs on real-world financial research tasks. arXiv preprint arXiv:2508.00828, 2025. URL https://arxiv.org/abs/2508.00828.
- [7]Noam Brown. Implications of large-scale test-time compute. https://x.com/polynomial/status/2064210146558136827, 2026. Post on X. Accessed 2026-10-04.
- [8]Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In EMNLP, 2021. URL https://aclanthology.org/2021.emnlp-main.300/.arxiv.org/abs/2109.00122
- [9]Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. Dated Data: Tracing knowledge cutoffs in large language models. In COLM, 2024. URL https://openreview.net/forum?id=wS7PxDjy6m.arxiv.org/abs/2403.12958
- [10]Jonathan H. Choi, Kristin E. Hickman, Amy B. Monahan, and Daniel Schwarcz. ChatGPT goes to law school. Journal of Legal Education, 71(3):387–400, 2022.DOI
- [11]Databricks. Introducing OfficeQA Pro V2: A new benchmark for enterprise grounded-reasoning. https://www.databricks.com/blog/introducing-officeqa-pro-v2-new-benchmark-enterprise-grounded-reasoning, 2026. Accessed 2026-10-12.
- [12]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026a. URL https://arxiv.org/abs/2606.19348.
- [14]Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In NAACL, 2024. URL https://aclanthology.org/2024.naacl-long.482/.arxiv.org/abs/2311.09783
- [15]Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In EMNLP, 2021. URL https://aclanthology.org/2021.emnlp-main.446/.arxiv.org/abs/2012.14913
- [16]GLM-5 Team. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. URL https://arxiv.org/abs/2602.15763.
- [17]Google. Introducing Gemini 3.8 Flash and 3.8 Flash Cyber. https://blog.google/innovation -and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/, 2026. Accessed 2026-10-04.
- [18]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhendan Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081): 633–638, 2025. URL https://doi.org/10.1038/s41586-025-09422-z.
- [19]Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, and Nima Mesgarani. LiveMathematicianBench: A live benchmark for research-level mathematical reasoning with proof sketches. In COLM, 2026. URL https://openreview.net/forum?id=XFwoUUJNpm.arxiv.org/abs/2604.01754
- [20]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
- [21]Masanori Hirano. Construction of a Japanese financial benchmark for large language models. In FinNLP-KDF-ECONLP Workshop at LREC-COLING, 2024. URL https://aclanthology.org/2024.finnlp-1.1/.arxiv.org/abs/2403.15062
- [22]Kwan Yuen Iu and Vanessa Man-Yi Wong. ChatGPT by OpenAI: The End of Litigation Lawyers? SSRN Scholarly Paper 4339839, 2023. URL https://papers.ssrn.com/abstract=4339839.DOI
- [23]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In ICLR, 2025. URL https://openreview.net/forum?id=chfJJYC3iL.arxiv.org/abs/2403.07974
- [24]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In ICLR, 2024. URL https://openreview.net/forum?id=VTF8yNQM66.arxiv.org/abs/2310.06770
- [25]Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. Evaluating GPT-4 and ChatGPT on Japanese medical licensing examinations. arXiv preprint arXiv:2303.18027, 2023. URL https://arxiv.org/abs/2303.18027.
- [26]Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270):20230254, 02 2024. URL https://doi.org/10.1098/rsta.2023.0254.
- [27]Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. URL https://arxiv.org/abs/2607.24653.
- [28]Maxim Massenkoff and Peter McCrory. Labor market impacts of AI: A new measure and early evidence. https://www.anthropic.com/research/labor-market-impacts, 2026. Accessed 2026-10-13.
- [29]Tatsuki Masuda, Kei Nakagawa, and Takahiro Hoshino. Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on auditing. JSAI Technical Report, Type 2 SIG, 2023(FIN-031):81–88, 2023.
- [30]Mathematical Association of America. American Invitational Mathematics Examination (AIME). https://maa.org/maa-invitational-competitions, 2026. Accessed 2026-10-04.
- [31]Mike A Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Lizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Kumar Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muenninghoff, Robert Kwesi Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Jenia Jitsev, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In ICLR, 2026. URL https://openreview.net/forum?id=a7Qa4CcHak.
- [32]Mistral AI. Introducing Mistral Large 4. https://mistral.ai/news/mistral-large-4, 2026. Accessed 2026-10-10.
- [33]Moonshot AI. Kimi K2.6. https://huggingface.co/moonshotai/Kimi-K2.6, 2026a. Accessed 2026-10-04.
- [34]Moonshot AI. Introducing Kimi K3.1 Pro. https://www.kimi.ai/blog/kimi-k3-1-pro, 2026b. Accessed 2026-10-24.
- [35]Kei Nakagawa, Masanori Hirano, and Kaito Takano. Survey and perspectives on large language models in Japanese financial sector. Jxiv preprint, 2025. URL https://jxiv.jst.go.jp/index.php/jxiv/preprint/view/1268/. In Japanese.
- [36]Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023. URL https: //arxiv.org/abs/2303.13375.arxiv.org/abs/2303.13375
- [42]Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins, Josh Joseph, Ivan Zhou, Xinyi Wang, Ashutosh Baheti, Owen Oertell, Sam Havens, Jacob Portes, Michael Bendersky, Erich Elsen, and Matei Zaharia. OfficeQA Pro: An enterprise benchmark for end-to-end grounded reasoning. In COLM, 2026. URL https://openreview.net/forum?id=NGrVhUG9Wl.arxiv.org/abs/2603.08655
- [43]Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In ICLR, 2024. URL https://openreview.net/forum?id=KS8mIv etg2.arxiv.org/abs/2310.17623
- [44]Long Phan, Alice Gatti, Nathaniel Li, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dan Hendrycks, Ziwen Han, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Aakaash Nattamai, Gordon McKellips, Anish Cheraku, Asim Suhail, Ethan Luo, Marvin Deng, Jason Luo, Ashley Zhang, Kavin Jindel, Jay Paek, Kasper Halevy, Allen Baranov, Michael Liu, Advaith Avadhanam, David Zhang, Vincent Cheng, Brad Ma, Evan Fu, Liam Do, Joshua Lass, Hubert Yang, Surya Sunkari, Vishruth Bharath, Violet Ai, James Leung, Rishit Agrawal, Alan Zhou, Kevin Chen, Tejas Kalpathi, Ziqi Xu, Gavin Wang, Tyler Xiao, Erik Maung, Sam Lee, Ryan Yang, Roy Yue, Ben Zhao, Julia Yoon, Xiangwan Sun, Aryan Singh, Clark Peng, Tyler Osbey, Taozhi Wang, Daryl Echeazu, Timothy Wu, Spandan Patel, Vidhi Kulkarni, Vijaykaarti Sundarapandiyan, Andrew Le, Zafir Nasim, Srikar Yalam, Ritesh Kasamsetty, Soham Samal, David Sun, Nihar Shah, Abhijeet Saha, Alex Zhang, Leon Nguyen, Laasya Nagumalli, Kaixin Wang, Aidan Wu, Anwith Telluri, Summer Yue, Alexandr Wang, Dmitry Dodonov, Tung Nguyen, Jaeho Lee, Daron Anderson, Mikhail Doroshenko, Alun Cennyth Stokes, Mobeen Mahmood, Oleksandr Pokutnyi, Oleg Iskra, Jessica P. Wang, John-Clark Levin, Mstyslav Kazakov, Fiona Feng, Steven Y. Feng, Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Serguei Popov, Robert Gerbicz, Geoff Galgon, Johannes Schmitt, Will Yeadon, Yongki Lee, Scott Sauers, Alvaro Sanchez, Fabian Giska, Marc Roth, Søren Riis, Saiteja Utpala, Noah Burns, Gashaw M. Goshu, Mohinder Maheshbhai Naiya, Chidozie Agu, Zachary Giboney, Antrell Cheatom, Francesco Fournier-Facio, Sarah-Jane Crowson, Lennart Finke, Zerui Cheng, Jennifer Zampese, Ryan G. Hoer, Mark Nandor, Hyunwoo Park, Tim Gehrunger, Jiaqi Gai, Ben McCarty, Alexis C. Garretson, Edwin Taylor, Damien Sileo, Qiuyu Ren, Usman Qazi, Lianghui Li, Jungbae Nam, John B. Wydallis, Pavel Arkhipov, Jack Wei Lun Shi, Aras Bacho, Chris G. Willcocks, Hangrui Gao, Sumeet Motwani, Emily de Oliveira Santos, Johannes Veith, Edward Vendrow, Doru Cojoc, Kengo Zenitani, Joshua Robinson, Longke Tang, Yuqi Li, Joshua Vendrow, Natanael Wildner Fraga, Vladyslav Kuchkin, Andrey Pupasov Maksimov, Pierre Marion, Denis Efremov, Jayson Lynch, Kaiqu Liang, Aleksandar Mikov, Andrew Gritsevskyi, Julien Guillod, Gözdenur Demir, Dakotah Martinez, Ben Pageler, Kevin Zhou, Saeed Soori, Ori Press, Henry Tang, Paolo Rissone, Sean R. Green, Lina Brüssel, Moon Twayana, Aymeric Dieuleveut, Joseph Marvin Imperial, Ameya Prabhu, Jinzhou Yang, Nick Crispino, Arun Rao, Dimitri Zvonkine, Gabriel Loiseau, Mikhail Kalinin, Marco Lukas, Ciprian Manolescu, Nate Stambaugh, Subrata Mishra, Tad Hogg, Carlo Bosio, Brian P. Coppola, Julian Salazar, Jaehyeok Jin, Rafael Sayous, Stefan Ivanov, Philippe Schwaller, Shaipranesh Senthilkumar, Andres M. Bran, Andres Algaba, Kelsey Van den Houte, Lynn Van Der Sypt, Brecht Verbek en, David Noever, Alexei Kopylov, Benjamin Myklebust, Bikun Li, Lisa Schut, Evgenii Zheltonozhskii, Qiaochu Yuan, Derek Lim, Richard Stanley, Tong Yang, John Maar, Julian Wykowski, Mart Oller, Anmol Sahu, Cesare Giulio Ardito, Yuzheng Hu, Ariel Ghislain Kemogne Kamdoum, Alvin Jin, Tobias Garcia Vilchis, Yuexuan Zu, Martin Lackner, James Koppel, Gongbo Sun, Daniil S. Antonenko, Steffi Chern, Bingchen Zhao, Pierrot Arsene, Joseph M. Cavanagh, Daofeng Li, Jiawei Shen, Donato Crisostomi, Wenjin Zhang, Ali Dehghan, Sergey Ivanov, David Perrella, Nurdin Kaparov, Allen Zang, Ilia Sucholutsky, Arina Kharlamova, Daniil Orel, Vladislav Poritski, Shalev Ben-David, Zachary Berger, Parker Whitfill, Michael Foster, Daniel Munro, Linh Ho, Shankar Sivarajan, Dan Bar Hava, Aleksey Kuchkin, David Holmes, Alexandra Rodriguez-Romero, Frank Sommerhage, Anji Zhang, Richard Moat, Keith Schneider, Zakayo Kazibwe, Don Clarke, Dae Hyun Kim, Felipe Meneguit ti Dias, Sara Fish, Veit Elser, Tobias Kreiman, Victor Efren Guadarrama Vilchis, Immo Klose, Ujjwala Anantheswaran, Adam Zweiger, Kaivalya Rawal, Jeffery Li, Jeremy Nguyen, Nicolas Daans, Haline Heidinger, Maksim Radionov, Václav Rozhoň, Vincent Ginis, Christian Stump, Niv Cohen, Rafal Poświata, Josef Tkadle c, Alan Goldfarb, Chenguang Wang, Piotr Padlewski, Stanislaw Barzowski, Kyle Montgomery, Ryan Stendall, Jamie Tucker-Foltz, Jack Stade, T. Ryan Rogers, Tom Goertzen, Declan Gradd, Abhishek Shukla, Alan Givré, John Arnold Ambay, Archan Sen, Center for AI Safety, Scale AI, and HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature, 649(8099):1139–1146, 2026. URL https://doi.org/10.1038/s41586-025-09962-4.
- [46]Qwen. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026. URL https://arxiv.org/abs/2608.30320.
- [47]Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko. PostTrainBench: Can LLM agents automate LLM post-training? In ICLR, 2026. URL https://arxiv.org/abs/2603.08640.
- [48]Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In EMNLP, 2020. URL https://aclanthology.org/2020.emnlp-main.437/.arxiv.org/abs/2002.08910
- [49]Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. When FLUE meets FLANG: Benchmarks and large pretrained language model for financial domain. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), EMNLP, 2022. URL https://aclanthology.org/2022.emnlp-main.148/.
- [50]Daniel Shepard and Robin Salimans. AutomationBench. arXiv preprint arXiv:2604.18934, 2026. URL https://arxiv.org/abs/2604.18934.
- [51]Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, Xiang Li, Xiangning Lin, Yaxuan Deng, Han Xu, Yuangang Li, Shanda Li, Zizhao Chen, Hanwen Xing, Harsh Raj, Bo Chen, Quan Shi, Steven Dillmann, Yipeng Gao, Puneesh Khanna, Ruofan Lu, Chao Beyond Zhou, Michael Yang, Robert Zhang, Siyuan Chai, Jiayu Chang, Yizhao Chen, Xiaokun Chen, Yiwei Dai, Wenting Yang, Hange Liu, Minghao Liu, Zihan Wang, Adnan El Assadi, Benedikt Stroebl, E. Kelly Buchanan, Han Meng, Junwei He, Longxuan Yu, Radin Shayanfar, Yukyung Lee, Zhikang Dong, Allen G Hart, Anjiang Wei, Anurag Kashyap, Arpandeep Khatua, Audrey Jixin Zheng, Chengrui Ma, David Heineman, Dubing Chen, Hai-Anh Trinh, Haishuo Fang, Hefan Zhang, Hui Shen, Issa Sugiura, Jiankai Sun, Jiechao Gao, Junhong Lin, Junnan Li, Kai Yang, Lei Hsiung, Maoyou Wang, Mengze Tang, Nabil Omi, Negin Raoof, Nicholas Edwards, Octavia Guo, Orfeas Menis Mastromichalakis, Pengliang Ji, Przemysław Hejman, Qi Qi, Qunshu Lin, Richard Zhuang, Rui Yang, Ruichen Zheng, Ryan Marten, Shaghayegh Fazliani, Shizheng Hou, Sicong Jiang, Sijie Li, Boqin Yuan, Michael Glass, Song Bian, Terry Yue Zhuo, Tianqing Wu, Tom Tang, Wanjia Zhao, Weihao Xuan, Wenhua Liang, Xian Liu, Xin Lan, Xuan Zhang, Xuandong Zhao, Yanchuan Tang, Yifan Jiang, Yijiang Li, Yitong Guan, Yizhi Li, Yonghui Liu, Yuheng Tang, Yujun, Mao, Yunfei Zhao, Yuxin Wang, Yuxuan Tang, Zhenheng Tang, Zhifei Li, Ziruo Wang, Ziyu She, Kaiyuan Liu, Jheb Chaabane, Yuxin Tang, Xiangyi Li, Satya Sai Srinath Namburi GNVV, Xinyue Zheng, Andy Konwinski, Boxuan Li, Leon Liangyu Chen, Alex Dimakis, Nicholas Carlini, Soroush Vosoughi, Sanmi Koyejo, Di He, Etash Guha, Benjamin Feuer, Mike Merrill, Ludwig Schmidt, and Alex Shaw. Harbor Adapters and Harbor-Index: Infrastructure and a curated meta-dataset for large-scale agentic evaluation. arXiv preprint arXiv:2609.04298, 2026. URL https://arxiv.org/abs/2609.04298.
- [52]Issa Sugiura, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, Taro Makino, Teppei Usuki, and Takashi Ishida. CoffeeBench: Benchmarking long-horizon LLM agents in heterogeneous multi-agent economies. In NeurIPS (Evaluations and Datasets Track), 2026a. URL https://openreview.net/forum?id=YlQnsbF9X.arxiv.org/abs/2606.16613
- [53]Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, and David Ha. EDINET-Bench: Evaluating LLMs on complex financial tasks using Japanese financial statements. In ICLR, 2026b. URL https://openreview.net/forum?id=Dxns0Cj15A.arxiv.org/abs/2506.08762
- [54]Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. URL https://arxiv.org/abs/2504.12516.
- [55]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddhartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A challenging, contamination-limited LLM benchmark. In ICLR, 2025. URL https://openreview.net/forum?id=SKYHBTaXvA.arxiv.org/abs/2406.19314
- [56]Xiaomi MiMo. MiMo-V2.6-Pro-RL. https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL, 2026. Accessed 2026-10-04.
- [57]Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, GUOJUN XIONG, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, Huang Jiajia, Xiao-Yang Liu, Alejandro Lopez-Lira, Benyou Wang, Yanzhao Lai, Hao Wang, Min Peng, Sophia Ananiadou, and Jimin Huang. FinBen: An holistic financial benchmark for large language models. In NeurIPS (Datasets and Benchmarks Track), 2024. URL https://openreview.net/forum?id=loDHZstVP6.
- [58]Yue Xun, Junyu Liu, Qian Niu, Xinyi Wang, Zheng Yuan, Zirui Li, Zequn Zhang, Bowen Zhao, Shujun Wang, Irene Li, Kan Hatakeyama-Sato, Yusuke Iwasawa, and Yutaka Matsuo. JMed48k: A multi-profession Japanese medical licensing benchmark for vision-language model evaluation. arXiv preprint arXiv:2605.22080, 2026. URL https://arxiv.org/abs/2605.22080.
- [59]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In NeurIPS, 2024. URL https://openreview.net/forum?id=mXpq6ut8J3.arxiv.org/abs/2405.15793
- [60]John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, and Ofir Press. ProgramBench: Can language models rebuild programs from scratch? arXiv preprint arXiv:2605.03546, 2026. URL https://arxiv.org/abs/2605.03546.
- [61]Yuhao Zhang, O. Ozan Koyluoglu, Thejas Venkatesh, Richard Diehl Martinez, Vishank Bhatia, Arash Alidoust, and Ashwin Paranjape. FrontierFinance: A challenging benchmark for measuring frontier intelligence of finance agents. arXiv preprint arXiv:2608.11683, 2026. URL https://arxiv.org/abs/2608.11683.
- [62]Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS (Datasets and Benchmarks Track), 2023. URL https://openreview.net/forum?id=ucchPGD1ao.arxiv.org/abs/2306.05685
