betaFeedback

WorksComputer Science

CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development

Abstract

We developed CAE-Bench, a benchmark to evaluate AI agents on Computer-Aided Engineering (CAE) solver development. A CAE solver is more than a PDE solver; it must be able to generalize across various inputs, such as different boundary conditions, geometries, and material properties. CAE-Bench evaluates the generalizability of AI-developed CAE solvers by providing the same task, such as a heat transfer problem, multiple cases, with each case specifying an input. The AI agent develops a solver for a fixed task using only a few public cases, and the resulting solver is then tested on held-out cases, i.e., cases it has never seen before. We evaluate 12 models across a range of reasoning settings on 80 computational tasks. Our experiments show that even if a solver developed by an AI agent performs well on the public cases, it may even get 0 score on held-out cases. For tasks such as fluid dynamics problems, even the solver developed by GPT-6 Astra performs poorly on held-out cases. A data format and an SDK for task registration are provided, so the researcher can use the benchmark on their own infrastructure. We hope that CAE-Bench will help the community to develop AI agents that can develop reusable CAE solvers that generalize to various inputs.

Introduction

For a PDE problem with given geometry and boundary conditions, it is not surprising that AI agents are capable of developing an accurate numerical solver. The AI-developed solvers for a fixed PDE problem can be evaluated through PDEAgent-Bench ([2]) or CFDLLMBench ([5]). However, for an industrial-level Computer-Aided Engineering (CAE) solver, solving a fixed PDE is not enough. The solver must be able to handle different geometries, boundary conditions, and material properties. The inputs of held-out cases should not be provided during development. Therefore, for a single task, the AI agent is required to develop a CAE solver that can generalize across cases with varying inputs. In this paper, we introduce CAE-Bench, a benchmark to evaluate the ability of AI agents to develop reusable CAE solvers that generalize across inputs.

We consider an extreme example. The task is to develop a solver for the governing equations of a 2-dimensional incompressible flow of a Newtonian fluid. The solver predicts velocity and pressure fields for given inputs. If the AI agent is only required to solve the task for fixed inputs, and the inputs are provided to the agent during development, then the agent can compute and store the answer during development, then return it during evaluation. This does not demonstrate that the agent can develop a reusable solver, since the resulting solver only works for these fixed inputs.

To evaluate the generalizability of AI-developed CAE solvers, we evaluate solvers on the same task with different cases, each case containing inputs and the reference used for scoring. All cases are distinguished into Public cases and Held-out cases. Public cases provide public inputs and example outputs during development. The example outputs have the required format, but their values are random. Held-out cases contain inputs that vary in geometry, material response, initial states, or boundary-condition types. These held-out cases are not provided to the agent during development. In summary, the public cases are used in both development and evaluation, while the held-out cases are only used in evaluation.

Besides input, each case is associated with reference results collected from analytical solutions, existing datasets, and established simulation software. These reference results are used for scoring during the evaluation phase, as discussed in § 2.

We also designed a data format to store the requirements of a given task, cases, and the output. Details are discussed in Section A. An SDK with tools for task registration and evaluation is also provided. Therefore, researchers can use the benchmark on their own infrastructure to evaluate the performance of their LLM model or harness.

The RC7 release of CAE-Bench contains 80 computational tasks, which include 13 prescribed-PDE tasks, 43 fluid-mechanics tasks, and 24 fixed-problem tasks adapted from CFDCodeBench (Somasekharan et al., [5]). Fixed-problem tasks do not have held-out cases and are reported separately. Replication of the complete evaluation is discussed in Appendix § 9.

Using the RC7 release, we evaluate 12 models across a range of reasoning settings. At medium effort, some AI-developed solvers receive full score on all public cases of a task. However, all AI-developed solvers got 0 score on at least 1 held-out case of the same task. Also, most 0-score cases reach numerical scoring, which means the solver runs successfully and the output format is correct. However, the output fails the accuracy requirements. These results show that the current best models are not capable of developing an industrial-level reusable CAE solver without human involvement.

Related work Several works have explored the ability of AI agents to develop numerical solvers. CodePDE (Li et al., 2025) uses debugging and iterative refinement to improve AI-developed PDE solvers. PDEAgent-Bench (Hang et al., [2]) evaluates whether AI agents can develop accurate enough PDE solvers using libraries such as DOLFINx, Firedrake, and deal.II. CFDLLMBench (Somasekharan et al., [5]) evaluates AI agents on CFD-related tasks, including code generation and OpenFOAM workflows. PDEBench (Takamoto et al., [6]) evaluates machine learning models by providing simulation datasets covering different initial conditions, boundary conditions, and PDE parameters. The reference solution is also provided to compare the machine learning based methods with traditional methods. These works mainly focus on developing solvers or completing simulations for given problems. For an industrial-level CAE solver, such complete information is impossible to gain during development.

Tasks and Evaluation Protocol

For a benchmark framework, the most important aspect is the clear definition of tasks and evaluation protocols (scoring). The detailed definition of tasks and cases is discussed in § 2.1 and the scoring rules are described in § 2.3. Besides definitions, we discuss the input data format in § 2.2 and the evaluation report format in § 2.3.

Tasks, Cases, and Observations

In this paper, the term task defines a physical or mathematical problem. By define, we mean that the task specifies the required inputs, outputs, scoring rules and execution requirements. A single task contains one or more evaluation cases. The term case defines a complete test example for the task. Each case contains specified inputs and reference results. Formally:

case=inputs+reference result.\text{case} = \text{inputs} + \text{reference result}.

The AI-developed solver must handle all the cases assigned to the task. Table 1 shows the task-case relationship in a released flow task.

AspectFixed by the taskSupplied by each case
AnalysisSteady, 2-dimensional, single-phase laminar flow; incompressible Newtonian fluidFluid domain; positive constant density and viscosity
ConditionsSupported mechanical boundary conditions, body forcing, and pressure conventionsCondition types, boundary assignments, values, and applicable pressure constraint
ObservationsDefinitions and units of velocity, pressure, and other supported quantitiesRequested quantities, locations, and point, average, or integral observations

Table 1. Cases for the 2-dimensional steady incompressible Newtonian-flow task.

The inputs describe the physical conditions, such as geometry, material properties, and boundary or initial conditions. During evaluation, the submitted solver receives the inputs and produces its own outputs.

Evaluation cases can be distinguished into two groups: Public cases and Held-out cases. Public cases provide inputs and template outputs (which means the data format is identical to the reference results but the numerical values are randomly generated) during development. Held-out cases are not accessible during development but only used for evaluation. A task may contain either or both case groups.

In this paper, we call each requested output produced by an AI-developed solver for a given case an observation. Each observation specifies a physical quantity, its units, and its form. An observation may contain values at multiple locations and times. For example, temperature is a physical quantity, while point values, cell averages, and spatial integrals are different forms of temperature observations. A temperature field can also be observed at a specific time or averaged over a period of time, resulting in different observations.

Data Formats and SDK

Each task contains instructions, DATA.md, and the shared CONTRACT.md as the agent’s initial prompt. DATA.md explains how to read the inputs and write the outputs. CONTRACT.md defines the common file formats.

In our implementation, an HDF5 file is used to store the sample identifiers, values, and field arrays. For sampling based on a mesh, the sample identifiers refer to node or element tags in a Gmsh .msh file. Each case input includes a problem.json file. problem.json specifies the physical setup, references the data files, and declares the requested outputs.

We also provide an optional SDK to help agents develop without having trouble understanding the data format. The SDK includes tools that test whether files meet format and compatibility requirements without providing evidence that shows numerical accuracy. Detailed discussions of the data format and the SDK are included in Appendix § 8.

These given prompts and tools guide the agent to develop solvers that follow the task’s physical assumptions and restrictions. Different tasks have different restrictions. For example, some tasks may require the solver to solve prescribed equations while other tasks let the agent decide the equations to solve.

The agent can use analytical expressions, numerical libraries, existing open-source software, or combine them, to develop a CAE solver. Therefore, CAE-Bench does not require Agents to implement every numerical method from scratch.

The submitted AI-developed CAE solver must include a solve.py entry point. CAE-Bench evaluates through the entry point for each case by

python solve.py -input problem.json -output result.h5

The submitted AI-developed solver is fixed after the development phase. The agent cannot revise the solver during the evaluation phase. For each case, CAE-Bench makes a fresh writable copy of the saved solver. The running solver receives the current case input. For that case, the solver must give the requested output. During evaluation, network access is not allowed. The solver must finish the case within the declared time, CPU, memory, and accelerator limits. Figure 1 shows the entire workflow of CAE-Bench including development, execution, and evaluation phases.

Figure 1. The workflow of CAE-Bench

Scoring Rules and Evaluation Report

Before giving any score, we first check the execution and output validity of AI-developed solvers. The solver receives 0 score in one case if it exceeds a time or resource limit, exits with an error, fails to produce output, or produces invalid output. CAE-Bench checks output identifiers, time axes, array shapes, and data types against the requirements. The output values must be finite and data types such as integer and Boolean must follow the exact agreement as in template outputs. Failure of a required type check sets the score to 0.

For floating-point observations, we compare the solver’s outputs with the reference values at all requested times and locations, including initial frames and boundaries. To calculate the score, we divide the outputs into groups based on the physical quantity, its components, and time. Each group contains all spatial values of one component at a given time, which are compared with the corresponding reference values.

For example, consider a case that requires 2-dimensional velocity and pressure fields at 10 different times. Since velocity has 2 components and pressure has 1 component, we have 20 groups for velocity and 10 groups for pressure, resulting in 30 groups in total. The number of spatial locations does not affect the number of groups, but only the number of values compared within each group.

We use a weighted normalized root-mean-square error (NRMSE) to evaluate each group. Let MM be the number of floating-point groups in a case, and let ii index the spatial locations within group jj. We use z^ji\hat{z}_{ji} and zjiz_{ji} to denote the solver’s output and reference value, respectively. The NRMSE for group jj is defined as

ej=∑iwji(z^ji−zji)2∑iwjimax⁡(∑iwji(zji−bj)2∑iwji, sj).e_j = \frac{\sqrt{\frac{\sum_i w_{ji}(\hat{z}_{ji}-z_{ji})^2}{\sum_i w_{ji}}}}{\max\left(\sqrt{\frac{\sum_i w_{ji}(z_{ji}-b_j)^2}{\sum_i w_{ji}}},\,s_j\right)}.

The default values of spatial weights wji>0w_{ji}>0 are equal for all ii and jj, and the default value for bjb_j is 0. The scale floor sj>0s_j>0 prevents division by zero. Spatial weights, scale floors, and scoring transformations are specified in the task descriptions and applied during evaluation. No further alignment or rescaling is allowed.

We denote the RMS of the MM group errors as EE and define the function p(E)p(E) that maps EE to a dimensionless score between 0 and 1. Formally,

E=1M∑j=1Mej2,p(E)={1,E≤0.05,(0.10−E)/0.05,0.05<E<0.10,0,E≥0.10.E = \sqrt{\frac{1}{M}\sum_{j=1}^{M}e_j^2}, \qquad p(E)= \begin{cases} 1, & E\le0.05,\\ (0.10-E)/0.05, & 0.05<E<0.10,\\ 0, & E\ge0.10. \end{cases}

Let T\mathcal{T} denote a task and NTN_{\mathcal{T}} be the number of cases in task T\mathcal{T}. Index kk runs from 1 to NTN_{\mathcal{T}} and identifies a case. The score of case kk is denoted by pkp_k. The case score pk=0p_k=0 if any required check fails. Otherwise, pk=p(E)p_k=p(E).

The task score PTP_{\mathcal{T}} is defined as the mean of all case scores (all assigned cases contribute, whether they are public or held-out cases). Formally,

PT=1NT∑k=1NTpk.P_{\mathcal{T}}=\frac{1}{N_{\mathcal{T}}}\sum_{k=1}^{N_{\mathcal{T}}}p_k.

For tasks with varying physical inputs, we report scores on public cases and held-out cases separately to evaluate how well the solver generalizes to unseen inputs. For fixed-problem tasks (tasks do not contain held-out inputs), the scores only reflect performance on the given inputs. Infrastructure tests are evaluated using exact-match or pass/fail checks, and their results are reported separately from computational tasks. Researchers can use adapters we provided to run the evaluation tools in their own environments.

Benchmark Construction

In this section, we discuss how to construct tasks and cases and how to obtain their reference results. Before discussing the details, we summarize the number of tasks and cases in CAE-Bench (RC7) in § 3.1. The design of cases for each task is described in § 3.2. How the reference results are obtained is discussed in § 3.3.

Benchmark Contents

As shown in Table 2, CAE-Bench (RC7) contains 2 infrastructure tests and 80 computational tasks. The 80 computational tasks are divided into 3 categories: prescribed-PDE tasks, fluid-mechanics tasks, and fixed-problem tasks.

Task categoryTasksEvaluation casesPublicHeld out
Prescribed PDEs132663263
Fluid mechanics4334043297
Fixed-problem tasks2424240
Computational total8063070560
Infrastructure tests217116

Table 2. Number of tasks and cases of CAE-Bench (RC7).

The prescribed-PDE tasks cover elliptic problems, transport, reaction and diffusion, and coupled systems. The fluid-mechanics tasks cover single-phase flow, energy transport, turbulence, interfaces, thin films, shallow water, rarefied gases, and fluids with memory. Both categories contain tasks with varying physical inputs.

The fixed-problem category uses 24 problems from CFDCodeBench [5]. Fixed-problem tasks do not have held-out cases.

The total 630 computational cases comprise 70 public cases and 560 held-out cases. All 24 cases in the fixed-problem category are public cases. The 2 infrastructure are used to check data handling and scoring behavior. In the final report, we exclude these infrastructure tests.

Case Design and Coverage

For the same task, we create various cases by changing material properties, geometry, boundary conditions, and initial conditions.

For example, the steady heat-conduction task (which is included in RC7) contains 32 cases with different geometries, thermal conductivities, heat sources, and boundary conditions. The boundary conditions include Dirichlet, Neumann, Robin, and mixed conditions. Among these cases, 4 use pure Neumann conditions with an additional mean-temperature constraint.

Figure 2 shows the input variations for another 2 tasks. The periodic advection task contains 24 cases, combining 8 velocities with 3 initial fields. The generalized-Newtonian flow task contains 8 cases with different material models and conditions. The material models include power-law, Cross, Carreau, and Carreau–Yasuda. Another task, transient viscoelastic flow, contains 9 cases with different initial conformation and wall-driving histories, including stopping and reversing wall motion.

Figure 2. Input variations for periodic advection and generalized-Newtonian flow tasks.

Reference Results

For each case, we get reference results from analytical solutions, manufactured solutions, existing simulation datasets, and existing simulation software. Table 3 shows how these reference results are collected for each task category.

Task categorySourceReference preparation
Heat conduction (2 tasks, 56 cases)Manufactured solutions and analytical seriesEvaluate the prescribed fields at the requested nodes and times.
PDEBench adaptations (11 tasks, 210 cases)Source trajectories and recomputations described in the accompanying recordsMatch the equations, initial states, time coordinates, units, and observations to the task.
Basic steady flow (2 tasks, 20 cases)Analytical solutions, Lethe, and ElmerEvaluate analytical fields or convert numerical solutions into the requested observations.
Other fluid-mechanics tasks (41 tasks, 320 cases)OpenFOAM, Elmer, SPARTA, PyClaw, Thetis, and formula evaluationEvaluate formulas or convert solver fields into the requested values, integrals, or statistics.
Fixed-problem adaptations (24 tasks, 24 cases)Analytical representations, NGSolve, Dedalus, and OpenFOAMApply the problem definitions described in the accompanying records. Export the requested observations.

Table 3. Reference sources and preparation for the computational suite.

The basic steady-flow cases use both analytical and numerical reference results. 8 basic 2-dimensional steady-flow cases use analytical solutions. 2 other basic 2-dimensional steady-flow cases use Lethe calculations. For the basic 3-dimensional steady-flow cases, we generate reference results with Elmer.

For the tasks borrowed from PDEBench ([6]), we use existing simulation datasets or recompute the solutions when necessary. For example, the Darcy task requires a steady solution, but the original dataset only provides states at finite times, so it requires a recomputation.

For the fixed-problem tasks adapted from CFDCodeBench, we compute reference results using the specified equations and boundary conditions. For example, in the cavity cases, the velocity boundary conditions are prescribed. The momentum and incompressibility equations determine the pressure up to an additive constant.

Experiments and Findings

In this section, we present the experimental results of CAE-Bench. We first describe the experimental setup in § 4.1. We then compare different models and reasoning-effort settings and discuss API costs in § 4.2, § 4.3, and § 4.4, respectively. The generalizability performance is discussed in § 4.5.

Experimental Setup

We test 6 OpenAI models, including GPT-5.6 Luna and GPT-6 Luna (each at none, low, medium, high, xhigh, and max), as well as GPT-5.6 Sol, GPT-6 Sol, GPT-6.1 Sol, and GPT-6 Astra (each at medium). We also test DeepSeek V4.1 Flash (high), Kimi K2.7 Code (thinking), Qwen3.8-Flash (low, medium, and xhigh), GLM-5.3-Flash (low, high, and max), Gemini 3.8 Flash (medium), and Gemini 4 Argon (medium). In total, we evaluate 12 models across 26 cases. Due to the so-called “security issue” under Anthropic dictatorial team ruled by Dario, we cannot access Claude-series model in China. Researchers can implement the experiment and submit into https://github.com/AXeonV/CAE-Bench

The mini-swe-agent 2.4.6 ([4, 7]) is used for all experiments. For the Gemini models, development and evaluation use local Harbor runs. For the other models, development and evaluation are conducted in separate Hugging Face Jobs. Agents have 30 to 120 minutes to develop a solver depending on the task. Each agent is given 8 vCPUs and 24 GiB of memory. If an agent fails to submit a solver, he receives a score of 0. More details are discussed in Appendix § 10.

We use the 80 computational tasks in RC7 to evaluate the models. Complete results are provided in Appendix § 10.

We calculate the score of each task by averaging its case scores, as defined in § 2.3. The final score is the average of the task scores, with equal weight for each task. All scores are reported on a 0-100 scale. We report the results for tasks with varying physical inputs and fixed-problem tasks separately.

Model Comparison

We compare 12 different models at medium or high reasoning effort. As shown in Table 4, GPT-6 Sol, GPT-6.1 Sol, and GPT-6 Astra achieve full scores on all 24 fixed-problem tasks. However, their average scores on tasks with varying physical inputs are only 39.90, 50.35, and 49.79, respectively. The result shows limited generalization performance. (More discussion in generalization in § 4.5.)

ModelReasoning effortVarying physical inputs (56)PDE (13)Fluid mechanics (43)Fixed-problem (24)Overall
GPT-5.6 Lunamedium29.3157.3920.8268.1140.95
GPT-6 Lunamedium36.0879.1323.0675.2247.82
GPT-5.6 Solmedium44.3787.2731.4097.2260.23
GPT-6 Solmedium39.9072.5830.02100.0057.93
GPT-6.1 Solmedium50.3583.7540.25100.0065.24
GPT-6 Astramedium49.7982.6939.85100.0064.85
Gemini 3.8 Flashmedium41.0675.2830.7198.5558.30
Gemini 4 Argonmedium48.2083.9137.4095.8362.49
DeepSeek V4.1 Flashhigh32.7354.5026.1591.6750.41
Qwen3.8-Flashmedium27.0068.5914.1375.0041.59
Kimi K2.7 Codethinking24.8558.9314.5476.2940.28
GLM-5.3-Flashhigh11.9230.776.2362.5027.10

Table 4. Model scores on the 80 computational tasks in RC7. For each model, we average case scores within each task and give equal weight to the evaluated tasks in each column. Scores are on a 0–100 scale.

All models perform better on prescribed-PDE tasks than on fluid-mechanics tasks. GPT-5.6 Sol achieves the highest score on prescribed-PDE tasks (87.27), while GPT-6.1 Sol achieves the highest score on fluid-mechanics tasks (40.25).

We further analyze why cases receive 0 score. As shown in Figure 3, for 6 OpenAI models, most 0-score cases are caused by scoring rejections rather than execution or output-format failures. For DeepSeek V4.1 Flash, Qwen3.8-Flash, Kimi K2.7 Code, and GLM-5.3-Flash, submission or execution failures occur more frequently.

Figure 3. Case outcomes across the 80 computational tasks in RC7.

Reasoning Effort Comparison

Figure 4 shows the scores of GPT-5.6 and GPT-6 Luna at 6 effort settings. The highest effort does not imply a higher score. For example, GPT-6 Luna achieves its highest score on tasks with varying physical inputs at xhigh (42.25), but the score drops to 38.18 at max. At max effort, the model fails to submit a solver for 17 tasks!

Figure 4. Comparison of reasoning-effort settings for GPT-5.6 Luna and GPT-6 Luna on 80 computational tasks. The tasks include 56 with varying physical inputs and 24 fixed-problem tasks. Each task has equal weight in the average score.

API Cost and Task Score

Figure 5 shows mean task score against mean recorded API cost per task. In general, higher reasoning effort leads to higher costs and higher scores, though this is not always the truth. Under our time limits,

Figure 5. Recorded API costs are in USD per task on a logarithmic scale.

higher reasoning effort can increase the time spent generating responses and running tools, leading to lower scores if the agent fails to submit a solver.

Generalization Performance

As shown in Figure 6(a), all models achieve lower scores on held-out cases than on public cases. For example, GPT-6.1 Sol achieves 52.21 on public cases but only 42.00 on held-out cases.

Figure 6. Public and held-out performance on tasks with varying physical inputs. (a) For tasks with both case types, we average public and held-out scores separately within each task and then average across tasks with equal weight. (b) Each bar shows the distribution of held-out case outcomes on tasks where every public case received full score. The numbers on the right give the held-out case count for each bar.

We also examine a more extreme situation, the tasks where the agents achieve full scores on all public cases. For instance, GPT-6.1 Sol achieves full scores on public cases for 22 tasks, but its average held-out score on these tasks is only 66.97. Similar results are observed for all models, as shown in Table 8. Therefore, the generalization performance of all models is limited.

ModelTasks with full score on all public casesMean held-out score (0–100)
GPT-5.6 Luna1557.35
GPT-6 Luna1760.98
GPT-5.6 Sol2259.16
GPT-6 Sol2060.13
GPT-6.1 Sol2266.97
GPT-6 Astra2267.44
Gemini 3.8 Flash2453.91
Gemini 4 Argon2165.56
DeepSeek V4.1 Flash1668.30
Qwen3.8-Flash1257.75
Kimi K2.7 Code1355.56
GLM-5.3-Flash481.78

Table 8. Mean held-out scores after full score on all public cases for the cases in Table 4.

Figure 6(b) shows how agents failed on held-out cases: most zero-score held-out cases are caused by scoring rejections while execution failure accounts for the rest.

Limitations

CAE-Bench mainly covers prescribed-PDE and fluid-mechanics tasks. The scores only reflect performance on the selected cases and do not guarantee that the developed solvers can handle all possible inputs or other CAE problems.

The reference results are collected from analytical solutions, existing datasets, and simulation software. However, not all reference results have been independently validated for numerical accuracy.

Although held-out cases are not provided during development, their source data may have appeared in model training data. Moreover, public and held-out cases differ in physical inputs and coverage. Therefore, the score differences cannot be attributed solely to withholding the cases. More details are provided in Appendix § 10.

Conclusion

We introduced CAE-Bench, a benchmark for evaluating the ability of AI agents to develop reusable CAE solvers. Unlike existing benchmarks that focus on fixed problems, CAE-Bench evaluates the generalization of AI-developed solvers across unseen physical inputs. Our experiments show that current AI agents are not capable of developing reusable CAE solvers without human involvement. Even when an AI-developed solver performs perfectly on cases used for development, it may fail on unseen cases. We hope CAE-Bench will help researchers develop AI agents capable of building reliable and reusable CAE solvers.

Acknowledgement

We thank the EIT High Performance Computing Platform for providing computational resources for this project. We also thank Wei Zhang of Southern University of Science and Technology for access to the Titan workstation. Many computations in our study would have been difficult without access to the workstation.

Data Format and Public Tools

Each task contains instruction.md, DATA.md, and the shared CONTRACT.md as the test materials provided to agents.

instruction.md is the overview information in the problem. It also gives the input data range. CONTRACT.md is the general data protocol for data storage and referencing. It serves as a common set of rules suitable for all problems. DATA.md defines the comprehensive rules for certain physical fields and constraints. Any specific rule not covered by CONTRACT.md is declared in DATA.md.

When the problem packages are being assembled, the problem.json will be generated for each case. The solver reads the input from problem.json. This file specifies the physical model, parameters and conditions. The information in fields and outputs describes the input fields and output requirements. Each term of declaration gives the details of HDF5 data such as dataset name, physical quantity, unit, data type, and component labels and shape. It may also specify the location and time axis, or explain whether the data are point values, averages or integrals. Conditions can be given as constants with units or refer to data stored in the input fields in an HDF5 file.

The input from an HDF5 file stores the concrete field array, entity id number and time value in the dataset. If there is a need to use a mesh, problem.json will also provide a Gmsh .msh file in MSH 4.1 ASCII format. The mesh file stores the nodes and edges information which is connected to the entity id number. Otherwise, the locations of this entity id number will be stored in the HDF5 file directly.

The axes of a field are in order: time axes, entity axes and component axes (declared in problem.json). Every position in the entity axes is aligned to the entity id number. The solver needs to write the output field to a separate HDF5 file and strictly ensure the order.

A Released Heat-Conduction Example

The public samples of the problem heat-conduction-steady-2d are in samples/01. The problem is designed to solve steady-state heat conduction on a unit square with no heat source. The thermal conductivity is 1 W/(m K). The temperature of 4 boundaries is 4 K. The mesh includes 9 nodes and 8 triangle units. Full information about the problem is available in supplement/heat-conduction-steady-2d-agent.zip.

The sample input/problem.json uses mesh.msh and input.h5. Here are the condition on the left boundary and the output requirement.

{
  "conditions": [
    {"role": "boundary", "variable": "temperature",
     "location": "left", "operator": "dirichlet",
     "data": {"coefficients": {"field": "left_coefficients"}}}
  ],
  "outputs": {
    "temperature": {
      "array": "temperature", "quantity": "temperature",
      "location": "nodes", "time_axis": "steady",
      "dtype": "float64", "kind": "scalar",
      "component_shape": [1], "components": ["value"],
      "unit": "K", "value_kind": "point"
    }
  }
}

The boundary condition uses left to refer to the left_ids dataset. It stores the element tags of the left boundary. The boundary condition also uses left_coefficients and gives the name to the input.h5 dataset accordingly. The dataset stores 5 temperature polynomial coefficients for each boundary in ascending power order. The polynomial in this case is always 4 K on the left boundary.

The output requirement uses nodes and steady to refer to node_ids and times in the input.h5 dataset. The shapes of the 3 datasets are (9,), (1,), and (1, 9, 1), respectively. temperature[0,i,0] gives the temperature in the time value 0 of node_ids[i]. This format is demonstrated by output/result.h5 using random temperature values.

Using the Data Tools

The solver uses -input to receive the path of problem.json and -output to receive the path of the HDF5 file it needs to write. The solver can also use SDKs provided by us to read and write. load_problem helps read and check the input. read_field helps read a dataset from the input HDF5 file according to the problem problem.json. write_result helps align and write the calculation output into the arrays. It also helps check the structure of the results under the rules of our DATA.md and CONTRACT.md.

The package also provides a validator to make structural checks. The validator is designed to reduce the burden of reading and writing data under our contract. We do not want the agents to have problems on understanding the dataset input or make errors on the output structure instead of the problem itself. The validator checks formatting, references, and declaration constraints. It does not verify or correct physical results.

Run the following commands in a Python environment to check the released example.

python -m pip install .
cd qbank/01-prescribed-pdes/1.1-elliptic/\
heat-conduction-steady-2d/samples/01
python -m caeoј validate-input input/problem.json
python -m caeoј validate-output --input input/problem.json \
  --output output/result.h5

validate-input checks the JSON declaration and mesh and HDF5 reference of problem.json. validate-output checks whether the data type and shape meet the requirements of the entity number id and time value.

python -m caeoj grade performs numerical grading. It grades using input cases and grading criteria grading.json on the solver.

Reference and Release Records

We list supplement files used in our experiments to help evaluate and reproduce our results.

Release Identity and Access

The tasks and test cases used in our experiments are available in RC7. The source repository is https://github.com/AXeonV/CAE-Bench. The supplement describes how to download and verify the release files.

The full dataset is available in CAE-Bench dataset repository on Hugging Face. Table 5 shows the detailed path and records.

File under supplement/Contents
reference_records.jsonReference data index and source
evidence/Record of reference data making process
experiment_results.jsonExperiment results and model configurations
heldout_losses.jsonHeld-out scores (tasks with full credit on all public cases)
score_group_diagnostics.jsonReport of each group (errors and checks)
score_sensitivity.jsonResult under different scoring rules
evaluation_manifest.jsonEvaluation file mappings, hashes, and resource limits
evaluation_overlay.zipModified and added evaluation files
evaluation_runtime/Evaluation runtime source code
heat-conduction-steady-2d-agent.zipPublic heat-conduction task package used in Appendix § 8.
resource_locations.jsonRecord of repository and dataset location
MANIFEST.jsonFile hash and size

Table 5. Supplementary files provided with CAE-Bench.

Public Packages and Evaluation Files

Each of our public/<task>-agent.zip includes our data format and public tools defined in Appendix § 8. qbank.zip contains reference data, grading criteria and some tools to create these questions. The sample data in public/<task>-agent.zip are not extracted from the reference data. The reference data are not visible to agents in experiments. To ensure the rigor of experiments, we halt the agents and grade their solver outside the environment independently.

Reference Sources and Evidence

supplement/reference_records.json records 82 tasks with 647 evaluation cases and 96 format examples. Reference data are associated with their sources and related information in the records above. These records also include how these data were generated and evaluated by us.

For the reference results, all heat-conduction cases use analytical solutions. The 2-dimensional steady-flow uses 8 analytical solutions and 2 Lattice calculations. The 3-dimensional steady-flow uses 10 Elmer generations. Others use the calculation results from OpenFOAM, Elmer, SPARTA, PyClaw, Thetis, and formula evaluation. supplement/reference_records.json records the settings used to generate these results and explains how we convert the source fields to the requested values. For the tasks adapted by PDEBench, some of them are directly extracted from the dataset while others are generated by us using the well-known solvers mentioned above. We do not guarantee that all of the reference data are credible enough. So we record the question creating process as detailed as possible in the related files.

Reconstruction Procedure and Limits

Rename the restored directory to rc7. Then run the command in a Python environment to check file integrity.

python scripts/rebuild_evaluation_snapshot.py \
--release-dir rc7 \
--manifest supplement/evaluation_manifest.json \
--overlay supplement/evaluation_overlay.zip --verify-only

supplement/evaluation_REPRODUCE.md explains how to extract a single task or the whole to a new directory and how to grade a new submitted solver. When grading, use the solver to generate a result.h5 from a certain case. Then use the grading command python -m caeoj grade to evaluate the result. The command itself does not check if the solver exceeds its runtime limits by judge-config.json.

Sensitivity to Scoring Choices

We test how different scoring rules affect the model comparison. We recompute the scores of the 12 cases in Table 4 on tasks with varying physical inputs using the saved numerical errors, without rerunning the solvers. All other scoring settings remain unchanged, and cases that failed mandatory checks still receive 0 score.

Table 6 compares 2 changes to the scoring rule. First, we change the full-score and 0-score thresholds by ±20%\pm20\% from the original (0.05, 0.10). Second, we replace RMS group aggregation with the maximum group error while keeping the original thresholds.

ModelRMS 4%/8%RMS 5%/10%RMS 6%/12%Max 5%/10%
GPT-5.6 Luna28.5229.3129.9927.88
GPT-6 Luna35.4136.0836.6034.16
GPT-5.6 Sol42.5644.3746.4638.83
GPT-6 Sol38.1639.9041.2234.43
GPT-6.1 Sol47.2650.3552.7043.01
GPT-6 Astra46.9349.7952.0542.77
Gemini 3.8 Flash39.6541.0642.4036.72
Gemini 4 Argon45.4848.2050.6140.63
DeepSeek V4.1 Flash31.5332.7333.5728.60
Qwen3.8-Flash26.3127.0027.5925.54
Kimi K2.7 Code24.6324.8525.0423.96
GLM-5.3-Flash11.8511.9211.9711.85

Table 6. Score sensitivity on tasks with varying physical inputs for the 12 cases in Table 4. Each entry averages the task scores with equal task weights. Scores use a 0–100 scale. We recompute the scores from stored group errors for the same submitted programs. Column headings give the full-score and zero-score thresholds, respectively. The bold column uses the official RMS 5%/10% scoring rule. Failures of execution, output-format checks, and exact-value checks retain 0 score.

Changing the thresholds does not affect the ranking of the 12 cases. Using the maximum group error reduces the scores and leaves the ranking unchanged.

Under the original RMS scoring rule, a case can receive full score even if some groups have NRMSE above 5%. This occurs in 173 of the 3,326 full-score case outcomes (5.2%).

Additional Experimental Results

The model comparison in § 4.2 uses one reasoning setting per model. Here we give the results for all 26 cases and describe how we ran the experiments and calculated costs.

Scores and Task Coverage

Table 7 shows the score and API cost under a certain effort of one model. Each row represents a valid report of tasks. The weight of each task in one task group is equal. If the evaluation process is done but the agent gives no submission, the score will be 0.

ModelEffortVarying physical inputs: ScoreVarying physical inputs: USD/taskFixed problems: ScoreFixed problems: USD/taskOverallReports
GPT-5.6 Lunanone26.250.019354.170.011034.6380/80
low25.800.013362.500.010336.8180/80
medium29.310.021668.110.015440.9580/80
high35.490.066695.770.035253.5880/80
xhigh39.670.1346100.000.054557.7780/80
max43.560.266896.110.079159.3280/80
GPT-6 Lunanone23.000.013654.730.007132.5280/80
low31.320.006062.500.004540.6780/80
medium36.080.018875.220.008847.8280/80
high41.190.035679.170.015352.5880/80
xhigh42.250.057089.240.018556.3580/80
max38.180.078591.670.034454.2280/80
GPT-5.6 Solmedium44.370.740797.220.407760.2380/80
GPT-6 Solmedium39.900.2523100.000.141157.9380/80
GPT-6.1 Solmedium50.350.5759100.000.149565.2480/80
GPT-6 Astramedium49.794.2573100.000.946264.8580/80
Gemini 3.8 Flashmedium41.061.344498.550.719458.3080/80
Gemini 4 Argonmedium48.203.790795.830.414462.4980/80
DeepSeek V4.1 Flashhigh32.730.216091.670.107250.4180/80
Qwen3.8-Flashlow30.160.179870.830.052042.3680/80
medium27.000.168975.000.066441.5979/80
xhigh16.830.137866.670.119831.7880/80
Kimi K2.7 Codethinking24.851.439776.290.580740.2880/80
GLM-5.3-Flashlow21.380.343473.550.110837.0380/80
high11.920.170562.500.124527.1080/80
max16.390.246366.670.130731.4780/80

Table 7. Scores and mean recorded API costs on RC7. Scores are on a 0–100 scale. The last column gives the number of valid task reports out of 80.

Qwen3.8-Flash has 79 reports at medium effort, with resolved-statistics-3d missing. Every other configuration has reports for all 80 tasks.

supplement/experiment_results.json gives the detailed results per task and case.

Development and Evaluation Conditions

In our experiments, the agent develops the solver in a public task workspace. It can use a shell to read and write files, run programs, and access the network.

For the Gemini models, development and evaluation use local Harbor runs. For the other models, development and evaluation run in separate Hugging Face Jobs. The jobs use 8 vCPUs and a 24 GiB memory allowance. The evaluator runs each case in a separate process without network access. The solver is required to produce result.h5 per task in between 4 s and 40 min. The time limit depends on each task.

During grading, the solver output result.h5 is compared to reference results. For most tasks, the grader has 60 s to judge per case. For the 3D compressible Navier–Stokes task, the solver has 30 min to produce result.h5 and the grader has 2 min to judge.

Held-out Performance After Full Public Score

For each model, Table 8 gives the number of tasks with full score on every public case and the mean score on their held-out cases. Only tasks with both public cases and held-out cases are included. Each task has equal weight.

API Cost Estimates

For each submission, we estimate the API cost by the recorded token counts. We use the provider’s rates for input and output tokens, including the rate for cached input where applicable. In our experiments, a run can be interrupted by the provider’s rate limit and be resumed. When a run resumes, we add the cost of the new requests to the cost recorded before it resumed. In Figure 5, we average scores and costs over the same tasks for each case.

We price DeepSeek usage at the peak rate. We convert CNY to USD at USD 0.150310793 per CNY using the reference rates for October 30, 2026 from the European Central Bank [1].

References

  1. [1]European Central Bank (2026). Euro foreign exchange reference rates. Rates for October 30, 2026. USD/EUR: 1.1172. CNY/EUR: 7.4326. Accessed November 1, 2026.
  2. [2]Hang, Z., Yashengjiang, Y., Li, J., Dong, H., Wei, Y., Hao, Z., Ma, J., Bai, S., Kai, H., Yue, X., Si, G., Jiang, D., Yao, C., Hu, Z., Zhang, J., Liu, P., Shen, Y., Ren, X., Liu, L., Xu, Z., Li, H., Yao, Q., Dong, H., and Wang, H. (2026). PDEAgent-Bench: A multi-metric, multi-library benchmark for PDE solver generation.arxiv.org/abs/2605.09636
  3. [4]Lieret, K. and Jimenez, C. E. (2026). mini-swe-agent. Software, version 2.4.6. Accessed November 1, 2026.
  4. [5]Somasekharan, N., Yue, L., Cao, Y., Li, W., Emami, P., Bhargav, P. S., Acharya, A., Xie, X., and Pan, S. (2026). CFDLLMBench: A benchmark suite for evaluating large language models in computational fluid dynamics. Journal of Data-Centric Machine Learning Research.arxiv.org/abs/2509.20374
  5. [6]Takamoto, M., Praditia, T., Leiteritz, R., MacKinlay, D., Alesiani, F., Pflüger, D., and Niepert, M. (2022). PDEBench: An extensive benchmark for scientific machine learning.arxiv.org/abs/2210.07182
  6. [7]Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K. R., and Press, O. (2024). SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2405.15793

Paper details

vlnc:W_b8bphx6f
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents