betaFeedback

WorksComputer Science

Parametric CAD Bench v3: Evaluating Agentic Parametric CAD Authoring from Text and Engineering Drawings

Abstract

Producing an executable CAD script is not sufficient to establish that an agent has satisfied an engineering design request. Evaluation must also consider the resulting geometry, specified dimensions, and native CAD modeling history. We present Parametric CAD Bench v3, a 100-task benchmark comprising 30 text-to-CAD creation tasks, 30 text-to-CAD create-and-edit tasks, and 40 tasks that reconstruct mechanical parts from engineering drawings. Agents operate in a pinned FreeCAD environment and submit native CAD artifacts to a separate programmatic verifier. The primary metric is mean continuous task reward, combining geometry similarity and specification consistency rather than binary task success. A September 24, 2026 snapshot covers 13 model-agent systems; Claude Code with Claude Opus 5.5 achieves the highest overall point estimate, 61.03%, and an image-to-CAD mean reward of 84.66%. These measurements describe complete systems, not isolated model capabilities. We release versioned tasks, evaluation configurations, and inspectable result artifacts.

Introduction

Computer-aided design (CAD) systems represent engineering intent as editable digital models. In a parametric CAD system, the artifact contains more than final geometry: a geometry kernel represents and manipulates curves, surfaces, topology, and solids; constraint solvers enforce dimensional and geometric relationships; and feature histories and dependency graphs record modeling steps so that changes to upstream parameters or features propagate to downstream geometry. These capabilities are fundamental to both professional CAD systems such as Siemens NX, CATIA, Creo, and SolidWorks, and open-source systems such as FreeCAD, which uses Open CASCADE Technology (OCCT) as its geometry kernel. By contrast, CadQuery is a code-centric parametric CAD library built on OCCT. Our benchmark instead requires native FreeCAD documents with an application-managed feature tree and document dependencies, rather than a CadQuery program or exported B-rep alone.

This distinction matters for evaluating CAD agents in real engineering workflows, where parametric CAD systems underpin modern mechanical design and manufacturing. An output can resemble the requested part while containing incorrect dimensions, missing features, broken dependencies, or a static shape in place of the required native feature structure. Conversely, alternative feature constructions can satisfy the same engineering requirements. Evaluating CAD agents therefore requires more than testing whether generated code executes or whether the resulting geometry is visually or geometrically similar.

Our central question is whether general-purpose AI agents can produce geometrically correct, specification-consistent, parametric CAD artifacts across three workflows.

The benchmark has evolved through three releases. We released Parametric CAD Bench v1 in May 2026 to evaluate text-based creation of native parametric FreeCAD models [11]. In September 2026, v2 retained the 100-task creation format while tightening validation and evaluation isolation: it upgraded the validator and FreeCAD runtime, moved held-out assets into a separate verifier image, and standardized reward reporting and failure handling [12]. V3 broadened the suite to include editing and reconstruction from engineering drawings [13]; the September 24 update added evaluations of Claude Opus 5.5 with two agent frameworks [14].

This paper focuses on v3; v1 and v2 provide historical context rather than additional evaluation cohorts.

Parametric CAD Bench v3 evaluates three workflows: text-to-CAD creation, text-to-CAD create-and-edit (two-stage creation and editing), and image-to-CAD (reconstruction from engineering drawings); we refer to the first two as creation and create-and-edit. Its native FreeCAD output contract permits inspection beyond a rendered image, while its grader combines geometric and task-specific specification checks. These checks are proxies for the intended engineering requirements; they do not by themselves certify manufacturing readiness or functional editability under arbitrary future modifications.

The contribution of this work is threefold:

  1. A versioned 100-task evaluation suite spanning three CAD workflows with native parametric output requirements.

  1. An isolated execution and programmatic scoring protocol, with frozen configurations and per-task diagnostic artifacts.

  1. A dated evaluation of complete model–agent systems, exposing differences across workflows and observed failure modes.

Related Work

CADGenBench: artifact-centered CAD evaluation. CADGenBench evaluates CAD generation and editing through tool-agnostic STEP/BREP submissions [4, 19]. Its public leaderboard accepts a ZIP archive of candidate artifacts; the hosted Hugging Face Space runs evaluation, and maintainers separately review methodology evidence before marking a submission validated [5, 19]. This is an artifact-submission protocol rather than a requirement to run a submitted agent inside a common task environment. However, it is not restricted to single-shot generation: its reference baseline is iterative, and evaluation tools are also available locally [4, 19]. Our emphasis is instead on evaluating complete agent executions in a versioned FreeCAD environment and requiring native parametric documents.

CADBench: multimodal CAD-program reconstruction. MIT’s CADBench evaluates reconstruction of executable, editable CAD programs from meshes and rendered images [7]. Its released evaluation pipeline currently accepts generated programs as CadQuery Python code; supporting other CAD programming languages would require adapting the pipeline [3]. It contains 18,000 samples across six benchmark families and five input modalities, with complexity-stratified STEP-based splits and diversity-aware sampling. Its six metrics cover volumetric IoU, surface IoU, Chamfer distance, valid-shape rate, token count, and operation count. Like Parametric CAD Bench v3, it evaluates executable CAD geometry rather than rendered appearance alone. However, its reported general-purpose VLM evaluation uses direct generation without tool use, self-correction, or multi-turn refinement; we evaluate complete interactive agent workflows in FreeCAD. CADBench emphasizes reconstruction from 3D observations and object renders, whereas our suite combines text-based creation, create-and-edit, and reconstruction from dimensioned engineering drawings. Its paper also distinguishes code editability from recovering the original feature tree, sketch constraints, or design intent, which it does not require. Our tasks instead require native FreeCAD documents with task-specific parametric structure and specification checks, without requiring exact reproduction of the reference construction history. The benchmarks are complementary: CADBench provides broad reconstruction and modality-shift diagnostics, while our smaller suite studies end-to-end CAD task completion by model–agent systems.

BenchCAD: programmatic CAD over CadQuery. BenchCAD 1.0 contains 17,900 execution-verified CadQuery programs across 106 industrial part families, with image-to-code, instruction-guided code editing, and visual/code question-answering tasks [21]. Its generation evaluation compares the executed geometry with a reference; the current website also reports agentic runs with rendering, measurement, and iterative tool use [2]. BenchCAD 2.0, released on October 12, 2026, after our September 24 snapshot, adds part and assembly construction from dimensioned engineering drawings, reverse engineering from rendered views, and agentic evaluation in an offline sandbox, all scored against reference geometry [1]. Its drawing-based part tasks are the closest counterpart to our image-to-CAD workflow. The distinction from Parametric CAD Bench v3 is therefore not whether agents can use tools, but the modeling interface and required output representation. CadQuery is a high-level parametric CAD library built on Python bindings to Open CASCADE Technology (OCCT), not the geometry kernel itself [6]. FreeCAD also uses OCCT, but provides a CAD application with native model history, geometry-constrained sketches, and a Python API [8]. BenchCAD’s CadQuery tasks emphasize parametric program synthesis, whereas our tasks require agents to construct and save native FreeCAD documents with the specified feature structure. Both are CAD workflows; our setting additionally exercises the application-level document and feature system.

CAD-bench: functional CAD generation. CAD-bench, presented at an ICML 2026 workshop, executes generated CAD code or agent-produced STEP files and checks the exported geometry with task-specific tests, including dimensional and pose verification, thread-profile analysis, and rigid-body simulation in Blender [20]. Its 17 tasks span four difficulty tiers, from basic solids to threaded mating pairs and gear trains. It therefore tests whether generated parts function together, whereas Parametric CAD Bench v3 requires native parametric FreeCAD documents across 100 tasks and scores their geometric similarity and compliance with named dimensions without simulating mechanisms. Despite the similar name, it is a separate benchmark with a different evaluation target.

3DCodeBench: mesh-based procedural 3D authoring. 3DCodeBench evaluates Blender Python scripts generated from text and images, including iterative execution and visual feedback [9]. Its evaluated outputs are meshes, assessed through executability, geometric and perceptual similarity, and pairwise human preferences. In contrast, Parametric CAD Bench v3 targets engineering-oriented mechanical CAD: native parametric FreeCAD documents with boundary-representation (B-rep) geometry and task-specific dimensional and structural checks. B-rep geometry represents surfaces and their topological relationships, rather than only a polygonal surface approximation [18]. Thus the distinction is between procedural mesh assets and dimensionally specified, editable CAD models, not between imprecise and precise software. 3DCodeBench includes manufactured objects, but its evaluation does not require native mechanical-CAD feature histories or engineering-dimension compliance. Likewise, our checks do not certify complete manufacturability.

Terminal-Bench and Harbor: agent evaluation infrastructure. Terminal-Bench provides complementary methodology for evaluating agents on difficult terminal workflows [17]. Our benchmark is built on Harbor, a framework for specifying and evaluating agent tasks in sandboxed environments [16]. We also contributed three FreeCAD tasks to Terminal-Bench 3.0/4.0: two create-and-edit tasks (freecad-impeller and freecad-spring-clip) and one image-to-CAD task (freecad-platform-drawing) [15]. These tasks follow the same create-and-edit and engineering-drawing reconstruction workflow types evaluated in Parametric CAD Bench v3.

Tasks and Dataset Construction

Task workflows

Table Table 1 describes the released task composition.

WorkflowTasksAgent inputRequired CAD artifacts
Text-to-CAD Creation30Part description and specified parametersNative parametric model
Text-to-CAD Create-and-Edit30Base description, edit instruction, and target requirementsBase and edited native models
Image-to-CAD40Engineering drawing and general execution requirementsNative model reconstructed from the drawing

Table 1. The v3 task composition. Text-to-CAD create-and-edit is a two-stage construction workflow, not editing an externally supplied CAD document.

In create-and-edit, the agent first constructs the base from text and then produces the edited model; both artifacts are graded. The task does not provide an externally authored starting FCStd. Scoring the two artifacts does not establish that the agent actually reused the base or preserved its feature history. Image-to-CAD tasks provide engineering drawings rather than the held-back dimensional specification.

Example tasks

Text-to-CAD creation examples. Two creation tasks illustrate the range of specifications. Figure 1 shows freecad-2002df3ce7, a trussed upper plate: a thin planar frame whose outline is the union of three rectangular sections with rounded corner chamfers, with mirrored trapezoidal recesses on the sides of the largest section. Diagonal truss webs separate seventeen triangular openings and one central six-sided opening, forming 18 internal loops; peripheral mounting holes and a center bore complete the part. Its 86 key parameters fall into five groups (Table 2). The description fixes the conventions needed to place them, including a sketch frame rotated about global +X+X so that layout +y+y points along global −Z-Z, and centroid, base-length, and base-angle definitions for each isosceles opening. The grader reference builds the plate from one sketch and one pad; it is one solid with 203 faces, 280 mm wide, 549 mm tall, and 8 mm thick.

Grader reference of a trussed upper plate in front oblique, front, and opposite oblique views

Figure 1. Grader reference of text-to-CAD creation example freecad-2002df3ce7, a trussed upper plate. (b) faces the plate, showing its 18 internal openings, side mounting holes, and centered end extension.

(a) Trussed upper plate, freecad-2002df3ce7
Parameter groupParametersWitnessCustomNative
Plate outline, thickness, and side recesses212019
Truss layout and diagonal angles7106
Mounting holes and center bore152013
Triangular and central openings413038
Second right-triangular opening2002
Total868078
(b) Asymmetric clevis joint connector, freecad-fa0a1abef3
Parameter groupParametersWitnessCustomNative
Central body3102
Large split-ring clevis10406
Clevis pin-hole cut2002
Fork prongs5104
Fork U-shaped opening5005
Side arm9009
Central relief slot3102
Upper and lower relief slots7007
Total447037

Table 2. Key-parameter groups of the two text-to-CAD creation examples and how the grader checks them. Witness: bound to a witness on the final geometry. Custom: a task-specific binding applied by param_check.py. Native: measured by param_check.py from native sketch or feature geometry.

Figure 2 shows freecad-fa0a1abef3, an asymmetric clevis joint connector: a rectangular central body with a large split-ring clevis at its +X+X end, a smaller fork at its −X-X end, and a long rounded side arm with three relief slots. The description states that the two ends are not mirror images and fixes global axes along, across, and through the connector. Its 44 key parameters fall into eight groups. The grader reference uses eight sketches, four pads, and four pockets; it is one solid with 31 faces, measuring 157 mm by 45 mm by 50 mm. Appendix § 9 reproduces both complete descriptions and parameter lists as given to the agent.

Grader reference of an asymmetric clevis joint connector in front oblique, axial, and opposite oblique views

Figure 2. Grader reference of text-to-CAD creation example freecad-fa0a1abef3, an asymmetric clevis joint connector. (b) looks along the split-ring axis, showing the fork, the large clevis, and the slotted side arm.

The held-back grader specification also states how each parameter is checked. Only 8 of the plate’s 86 parameters and 7 of the connector’s 44 are bound to witnesses on the final geometry; the task’s param_check.py measures the rest from native sketch and feature geometry. Neither task uses task-specific bindings. A candidate therefore earns specification credit only when its model exposes the named dimensions in a form these checks can measure. § 5.5 reports how the evaluated systems fared on both tasks.

Text-to-CAD create-and-edit example. Figure 3 illustrates freecad-ff7883ab0c. The agent first builds an axisymmetric flanged inlet housing: a continuous 50.8 mm central bore, a tapered inlet rim, a tubular middle body, a tapered rear hub, and a plain 152.4 mm rear flange disk. The edit adds a fourfold-symmetric front plate, 16.51 mm thick, whose outline joins side, corner, and transition arcs with short tangent connectors. It also adds four 13.34 mm through bolt holes in the plate and four 19.05 mm openings cut 34.54 mm deep from the rear flange face, each pattern placed from a signed seed center at uniform angular spacing. The tubular body, inlet rim, rear hub, rear disk envelope, and central passage must remain unchanged. Fifteen base parameters and 14 additional target parameters specify diameters, axial positions, arc radii, hole sizes and counts, and seed-center offsets. The complete supplied base description, edit instruction, and parameter lists appear in Appendix § 10.

Four views of the base and edited target

Figure 3. (a) Base: oblique view (b) Base: axial view (c) Edited target: oblique view (d) Edited target: axial view. Text-to-CAD create-and-edit example freecad-ff7883ab0c: trusted base reference (top) and edited target reference (bottom), shown in oblique and axial views. Each view uses the same angle and scale for both stages. The edit adds the front flange plate, its four bolt holes, and four rear-face openings while retaining the axisymmetric body and central bore.

This task couples an axisymmetric base with a non-circular, arc-bounded plate and two four-hole patterns on opposite faces. The script must save both answer_base.FCStd and answer_edit.FCStd, each as an editable single-body PartDesign model. Both stages are evaluated; a plausible final silhouette alone does not demonstrate correct base construction or compliance with the plate-outline and hole requirements. As with the other create-and-edit tasks, the two saved artifacts do not by themselves prove that the target reused the base feature history.

Image-to-CAD example. Figure 4 shows the input drawing and trusted reference for freecad-9749149002. The drawing combines orthographic, section, and isometric views of a linkage arm with two eye ends, angled transitions, and ribbed connecting beams. Dimension labels specify its 56 mm overall length, 19.5 mm depth, unequal 3 mm and 2 mm eye holes, and local radii and offsets. The agent must reconcile these views and dimensions to construct an editable single-solid PartDesign model.

Engineering drawing and oblique reference model of a front-wheel linkage arm

Figure 4. Image-to-CAD example freecad-9749149002, a front-wheel linkage arm. Top: the engineering drawing supplied to the agent, cropped to its frame; the scale line and general-tolerance note outside the frame are not shown. Bottom: an oblique view of the held-back grader reference.

Unlike the text-based examples, the agent receives no separate prose description or key-parameter list: the drawing is its only geometry input. It must write /app/answer.py and save /app/answer.FCStd under FreeCAD 1.1.0. The reference shown here is held back during evaluation; it illustrates the intended output, not an additional agent input.

Reference-model statistics

We count solid-modeling steps using FreeCAD’s native feature type hierarchy: objects derived from Part::Feature, excluding body containers, sketches, and datum helpers. Intermediate and hidden feature-history objects are included; a pattern counts as one feature regardless of instance count. Generic Part::Feature objects count too, so this statistic alone does not certify parametric editability. Sketch counts are reported separately. The main distribution gives each task one observation, using the target reference for create-and-edit rather than counting both stages as separate tasks. The 100 final reference models contain 687 solid-modeling feature objects in total, with a mean of 6.87, a median of 6, and a range of 1–20 features per task. For the 30 create-and-edit pairs, the base and target means are 2.47 and 5.60, respectively. These describe the provided reference constructions; they do not establish a minimum number of features required for a valid alternative solution or a calibrated measure of task difficulty.

WorkflowTasksMean featuresMedianMinMax
All final references1006.876120
Text-to-CAD Creation305.204120
Text-to-CAD Create-and-Edit (target)305.605.5210
Image-to-CAD409.088220

Table 3. Measured solid-modeling feature counts in the v3 grader references. Text-to-CAD create-and-edit contributes its target only to the 100-task distribution.

Figure 5 shows unit-bin histograms for the full benchmark and each workflow. Supplementary figures show feature-type composition, sketch counts, and base/target distributions for the 30 edit pairs.

Parametric CAD Bench v3: reference-model statistics. Four histograms: All tasks (one final reference each) (n=100), mean=6.87; Text-to-CAD Creation (n=30), mean=5.20; Text-to-CAD Create-and-Edit (target) (n=30), mean=5.60; Image-to-CAD (n=40), mean=9.08. Axes are Solid-modeling features and Number of tasks.

Figure 5. Solid-modeling features per final reference model. The aggregate contains 100 tasks, not 130 documents; dashed lines mark cohort means. Counts reflect reference construction choices rather than a calibrated difficulty measure or a required solution sequence.

Execution and Evaluation

Pinned runtime and isolation

The runtime uses FreeCAD 1.1.0, Harbor 0.23.0, and task schema 1.3. The verifier pins gnucleus-freecad-validator 0.6.0 with its v2 scorer. The benchmark version, validator package version, and scorer version are distinct identifiers. Reference geometry and task-specific grader assets are placed in a separate verifier environment rather than the agent environment. Each public job records its agent version and concurrency.

Programmatic scoring

The verifier computes two independent axes, geometry_similarity and cad_spec_consistency, using the default v2 scorer in gnucleus-freecad-validator 0.6.0 [10].

Geometric similarity. The scorer compares the candidate and reference solids using surface-type areas, volume, total surface area, normalized principal moments of inertia, and face-center iterative closest point (ICP) alignment. Let qsurfq_{\mathrm{surf}}, qvolq_{\mathrm{vol}}, qareaq_{\mathrm{area}}, qmomq_{\mathrm{mom}} and qICPq_{\mathrm{ICP}} denote their respective subscores in [0,1][0, 1]. After the gates below pass, the geometric score is

p=0.05qsurf+0.175qvol+0.175qarea+0.10qmom0.50,(1)p = \frac{0.05q_{\mathrm{surf}} + 0.175q_{\mathrm{vol}} + 0.175q_{\mathrm{area}} + 0.10q_{\mathrm{mom}}}{0.50}, \tag*{(1)}
g=p(0.50+0.50qICP).(2)g = p(0.50 + 0.50q_{\mathrm{ICP}}). \tag*{(2)}

Volume receives full credit at relative error at most 0.1% and zero subscore at 1% or more; the corresponding thresholds for surface-type areas, total surface area, and normalized principal moments

are 1% and 10%. Intermediate errors receive logarithmically interpolated credit. Surface-type comparison uses the largest per-type relative area error, with a denominator floor of 1% of the larger total surface area. An independently measured OCCT oriented bounding box is a gate, not an additional weighted subscore: maximum relative error across sorted extents of 10% or more sets g=0g = 0. Structural gates also require one non-empty PartDesign::Body producing one solid through allowed native features rather than baked shape holders. Candidates with more than 5,000 faces, or face/vertex counts differing from the reference by more than 50% of the larger count, are rejected. Gate failures override the spatial multiplier’s nonzero floor. Face-center alignment depends on face decomposition, so congruent geometry built through a different feature history need not receive g=1g = 1.

CAD specification consistency. The scorer parses the task’s held-back specification and checks its parameters against measurements extracted from the candidate document. CAD engineers wrote the parameter-checking logic shipped in each task’s tests/grader/param_check.py. Most tasks share a workflow-level implementation, and a subset adds case-specific checks. These checks encode ways to measure the dimensions, the correct placement of dimensions on the CAD part, feature relationships, and derived quantities required by the task’s specification. They supplement generic measurements with engineering-specific checks beyond global shape similarity. The v2 path honors trusted geometry bindings and case-local param_check.py refinements where supplied; remaining checks use generic per-kind measurements. The default scalar relative tolerance is 1%, with case-specific measurement logic and binding tolerances where applicable. Let nn be the number of parsed specification parameters and ff the number classified as inconsistent or not found. The default v2 failure budget is 10:

s={0,n=0,max⁡(0,1−fmin⁡(n,10)),n>0.(3)s = \begin{cases} 0, & n = 0,\\ \max\left(0,1-\frac{f}{\min(n,10)}\right), & n > 0. \end{cases} \tag*{(3)}

For n≥10n \ge10, each failed or unmeasurable parameter costs 0.1, preventing large specifications from diluting failures. This score measures the encoded checks, not every possible engineering requirement or functional editability under arbitrary future changes.

Combined task reward. Let g,s∈[0,1]g,s \in[0,1] denote geometry and specification scores. Define

H(g,s)={0,g=0 or s=0,2gsg+s,otherwise.(4)H(g,s) = \begin{cases} 0, & g = 0 \text{ or } s = 0,\\ \frac{2gs}{g+s}, & \text{otherwise}. \end{cases} \tag*{(4)}

For creation and image-to-CAD, the task reward is r=H(g,s)r = H(g,s). For create-and-edit, each output is scored separately, then combined:

redit=H(H(gbase,sbase),H(gtarget,starget)).(5)r_{\mathrm{edit}} = H\left(H(g_{\mathrm{base}},s_{\mathrm{base}}), H(g_{\mathrm{target}},s_{\mathrm{target}})\right). \tag*{(5)}

Thus a zero score on a required axis or deliverable cannot be rescued by another axis. The wrappers clamp rewards to [0,1][0,1]; text reward files use six decimal places, and the image-to-CAD wrapper also rounds its JSON reward to six decimal places.

The reported overall percentage is

S=100N∑i=1Nri,N=100.(6)S = \frac{100}{N}\sum_{i=1}^{N} r_i,\qquad N = 100. \tag*{(6)}

Final exception outcomes contribute zero. Workflow percentages are computed within each cohort, so S=0.30Screate+0.30Sedit+0.40SimageS = 0.30S_{\mathrm{create}} + 0.30S_{\mathrm{edit}} + 0.40S_{\mathrm{image}}.

This continuous reward is not Pass@1. We additionally report the fraction of final rewards exactly equal to 1.0 as the operational perfect-task rate. A scored trial is one without a final exception; scored count and positive-reward count are different quantities.

Results and Analysis

Dated leaderboard snapshot

Figure 9–Figure 12 visualize the 13-row September 24, 2026 snapshot. The plots are generated from the published leaderboard rows. The overall chart shows 95% intervals computed as the mean reward ±1.96s/100\pm1.96s/\sqrt{100}, where ss is the sample standard deviation of a system’s 100 per-task rewards. They describe uncertainty over the task sample, given one final trial per task, not variability across repeated runs of a system. Workflow charts show point estimates only.

Parametric CAD Bench v3 overall mean continuous task reward chart

Figure 9. Overall mean continuous reward across 100 tasks in the September 24, 2026 snapshot. Whiskers show 95% task-level intervals. Every final exception counts as zero. Ranking is by point estimate, not demonstrated statistical separation.

Parametric CAD Bench v3 Image-to-CAD mean continuous task reward across 40 engineering-drawing tasks

Figure 12. Image-to-CAD mean reward across 40 engineering-drawing tasks. Bars show workflow point estimates only. Different workflow populations prevent treating cross-workflow differences as a controlled modality ablation.

Claude Code with Claude Opus 5.5 achieves the highest overall point estimate, 61.03%, followed by mini-swe-agent with the same model at 57.96%, Codex with GPT-6 Astra at 56.87%, and Claude Code with Claude Fable 5.1 at 56.75%. Point-estimate ordering alone does not demonstrate a statistically decisive ranking.

The workflow leaders differ: Fable 5.1 with Claude Code has the highest creation point estimate (55.48%), Astra with Codex leads create-and-edit (44.37%), and Opus 5.5 with Claude Code leads image-to-CAD (84.66%). Because workflows use different task populations, these differences cannot be attributed solely to input modality.

Same-model and same-agent comparisons

We pair final rewards by task ID, rather than comparing only cohort means. For the following descriptive counts, two outcomes are similar when their rewards differ by at most 0.05 (5 percentage points); a higher outcome must exceed this tolerance. Similarity is not a statistical equivalence claim, nor does it imply success: two zero-reward outcomes also count as similar. Final exception outcomes retain their zero rewards.

Opus 5.5 versus Opus 5 with Claude Code. Within Claude Code, the overall point estimate rises from 50.86% for Opus 5 to 61.03% for Opus 5.5, a 10.18 percentage-point increase. The release analysis reports a paired 95% confidence interval of 5.22–15.13 points for this improvement [14]. It is a normal approximation over the 100 per-task reward differences: their mean ±1.96sd/100\pm1.96s_d/\sqrt{100}, where sds_d is their sample standard deviation. Table 4 shows that this is not a uniform improvement across workflows. For creation, Opus 5.5 is higher on 8 tasks, similar on 18, and lower on 4; for create-and-edit, the corresponding counts are 4, 23, and 3. Image-to-CAD shows the broadest improvement: Opus 5.5 is higher on 22 of 40 tasks, similar on 15, and lower on only 3. Its median paired drawing-task gain is 7.62 points, and all five drawings receiving zero reward under Opus 5 receive positive reward under Opus 5.5. The 22.02-point image-to-CAD mean gain accounts for approximately 87% of the overall improvement after applying the benchmark’s 30/30/40 workflow weights.

WorkflowNNA meanB meanA higherSimilarB higher
Text-to-CAD Creation3050.87%49.48%8184
Text-to-CAD Create-and-Edit3039.69%36.52%4233
Image-to-CAD4084.66%62.64%22153
Overall10061.03%50.86%345610

Table 4. Claude Code held constant; A: Opus 5.5, B: Opus 5. Higher means a task reward advantage greater than 5 percentage points; Similar includes the boundary. These counts do not establish statistical equivalence.

Claude Code versus mini-swe-agent with Opus 5.5. To support multimodal tasks, we add an explicit system prompt to the mini-swe-agent run: when a task references an image, the agent must first send that image through its multimodal observation channel, inspect the resulting visual input, and only then proceed with the task. This is paired with a configured bridge that converts marked, base64-encoded shell output into image content for the model; the prompt alone does not enable image transport. The Opus 5.5 configuration uses mini-swe-agent 2.4.6, preserves complete image payloads, and leaves malformed image output as text.

With Opus 5.5 held constant, Claude Code exceeds mini-swe-agent by 3.08 points. However, the paired interval computed the same way is [−1.15, 7.31] points, so these results do not establish an overall advantage for either agent. The workflow-level comparison in Table 5 is more informative than the overall ordering. Claude Code leads creation by 8.07 points, with 9 higher outcomes, 20 similar outcomes, and 1 lower outcome. mini-swe-agent leads create-and-edit by 1.68 points: Claude Code is higher on 2 tasks, similar on 24, and lower on 4. For image-to-CAD, 30 of 40 outcomes are similar, and each agent is higher on 5 tasks. Claude Code’s 2.90-point mean advantage therefore reflects the magnitudes of the differing outcomes, rather than a greater number of task-level wins. Both systems exceed 80% mean reward on drawing reconstruction with direct image input. These comparisons remain conditional on the evaluated configurations and dates, including different Claude Code versions; they are not completely controlled model or agent ablations.

WorkflowNNA meanB meanA higherSimilarB higher
Text-to-CAD Creation3050.87%42.79%9201
Text-to-CAD Create-and-Edit3039.69%41.38%2244
Image-to-CAD4084.66%81.77%5305
Overall10061.03%57.96%167410

Table 5. Opus 5.5 held constant; A: Claude Code, B: mini-swe-agent with multimodal bridge. Higher means a task reward advantage greater than 5 percentage points; Similar includes the boundary. These counts do not establish statistical equivalence.

Initial takeaway. The clearest observed gain is in engineering-drawing reconstruction: Opus 5.5 improves Claude Code’s image-to-CAD score by 22.02 points, versus 1.39 points for creation and 3.17 points for create-and-edit. Both Opus 5.5 systems exceed 80% on image-to-CAD (84.66% and 81.77%), so the strong drawing results are not confined to one agent implementation [14]. This suggests substantial progress in drawing-to-CAD task completion, but not uniformly reliable CAD authoring: the same systems reach only 39.69% and 41.38% on create-and-edit. These are distinct task cohorts, not a controlled test of modality alone. Together with the launch post’s Gemini comparison across two agents, the results also indicate that model capability and agent interface both matter; they do not identify a universally superior agent [13].

Multimodal CAD performance

Image-to-CAD produces the largest separation between the evaluated model–agent systems. Across the 13 rows, its mean reward ranges from 8.18% to 84.66%, a 76.49-point spread. By comparison, creation ranges from 23.32% to 55.48% (32.16 points), and create-and-edit from 23.23% to 44.37% (21.14 points). The five highest image-to-CAD systems achieve 62.64–84.66%, whereas the remaining eight achieve only 8.18–31.10%. Drawing reconstruction therefore distinguishes these systems much more sharply than either text-based workflow.

This pattern is also visible within a single agent family. Across the three Claude Code rows, image-to-CAD ranges from 62.64% with Opus 5 to 84.66% with Opus 5.5, a 22.02-point difference. The corresponding spreads are only 6.00 points for creation and 3.17 points for create-and-edit. Agent configuration matters as well: Gemini 3.8 Flash achieves 31.10% on image-to-CAD with mini-swe-agent’s multimodal bridge, versus 17.45% with Antigravity, a 13.65-point difference. Its creation and create-and-edit differences between these agents are smaller, at 4.66 and 5.90 points, respectively.

These results suggest that multimodal CAD task completion is a major differentiator among the evaluated systems. Strong drawing performance requires interpreting views, dimensions, and geometric relationships and translating them into a valid parametric model. However, these scores measure that entire workflow, not visual perception in isolation: CAD construction, tool use, image transport, and failure recovery can all affect the outcome. The larger score spread does not by itself establish which of these factors causes the gap, particularly because the workflows contain different tasks.

Efficiency

Cost per score point. We divide the known API cost of the complete 100-task run by its overall score on the 0–100 scale. API costs come from the usage recorded in each job’s artifacts. For Grok Build with Grok 4.7 and for mini-swe-agent with Kimi K3, Muse Spark 1.3, and Claude Opus 5.5, whose artifacts record no or incomplete cost, they come from our audits, which price the token usage in the saved trajectories at standard API rates. This descriptive ratio measures expenditure per achieved score point, not the marginal cost of improving a system by one point. Codex with GPT-6 Astra has the lowest ratio among all 13 systems: $317.81 divided by 56.87 points, or $5.59 per point. Its total spend is not the lowest, but the additional expenditure buys substantially higher reward: compared with GPT-5.6 Sol using Codex, Astra costs only $25.82 more while scoring 26.89 points higher.

Per-task speed. Astra also has the shortest mean elapsed trial duration, 14.42 minutes, compared with 16.58 minutes for Sol, 31.11 for Claude Code with Fable 5.1, and 32.51 for Claude Code with Opus 5.5. These are final-trial elapsed times, including setup, agent execution, and verification, rather than isolated model inference latency. We do not use whole-job wall time to rank speed, because concurrency, pauses, retries, and scheduling differ between runs.

Reward–cost trade-off. Astra’s 56.87% mean reward is close to Fable 5.1’s 56.75%, but its known API cost is approximately 70% lower and its mean trial duration is less than half as long. The higher-scoring Claude Code–Opus 5.5 system improves the point estimate by 4.17 points over Astra, at 1.81 times the API cost and 2.25 times the mean trial duration. Astra therefore offers a favorable observed balance of reward, cost, and per-task speed, rather than the highest reward at any price. These comparisons describe the evaluated configurations and their audited API accounting; they exclude some compute and hosting costs and do not establish why one system requires less time or expenditure.

Observed task difficulty

Across the 1,300 selected final trials from the 13 evaluated systems, 393 (30.23%) earn zero reward. Table 6 summarizes the pattern by workflow. Create-and-edit has the largest zero-outcome rate, 41.54%, compared with 22.56% for creation and 27.50% for image-to-CAD. These counts use the official final outcomes, including zero-valued final exceptions and reruns of provider or platform failures that produced no model outcome, rather than the superseded attempts.

WorkflowZero outcomesAll 13 zero≥7 zeroAny perfect
Text-to-CAD Creation22.56%056
Text-to-CAD Create-and-Edit41.54%5113
Image-to-CAD27.50%086
Overall30.23%52415

Table 6. Observed difficulty across 13 evaluated systems. Zero outcomes is the fraction of final trials with zero reward. The last three columns count distinct tasks: zero in all 13 systems, zero in at least 7 systems, and reward exactly 1.0 in at least one system. Cohorts contain 30 text-to-CAD creation, 30 text-to-CAD create-and-edit, and 40 image-to-CAD tasks.

Difficulty is concentrated in a subset of tasks. Twenty-four tasks receive zero reward from at least 7 of the 13 systems: 5 creation tasks, 11 create-and-edit tasks, and 8 image-to-CAD tasks. Five tasks receive zero from every system, all in create-and-edit. On 26 tasks, no system reaches 0.5: 9 creation, 16 create-and-edit, and 1 image-to-CAD. At the other end, 15 tasks earn a perfect reward of exactly 1.0 from at least one system (6 creation, 3 create-and-edit, and 6 image-to-CAD). Thus, strong aggregate drawing performance coexists with substantial remaining difficulty, especially in two-stage CAD construction. These are descriptive outcomes for the evaluated systems and configurations, not estimates of a task’s inherent solvability.

The saved diagnostics also show that zero reward is usually associated with the CAD scoring criteria rather than a final execution exception. Of the 393 zeros, 326 involve a zero geometry component, including 50 that also have a zero specification component. Another 33 fail only specification consistency; the remainder comprise 23 final exceptions, 7 missing outputs, and 4 scorer crashes. This breakdown describes observed failure surfaces, not independently adjudicated root causes.

Example task outcomes. The two creation examples fail in different ways. The trussed plate (Figure 1) is one of the 24 tasks with zero reward from at least 7 systems: 9 of the 13 systems receive zero (Table 7). Eight of these zeros come from the specification axis alone. These outcomes have geometry similarity between 0.126 and 0.445, but 12 to 80 of the 86 parameters are inconsistent or not found, which exhausts the failure budget of 10 in Equation (3). The ninth zero is a final exception. Among the four positive outcomes, Claude Fable 5.1 with Claude Code is nearly exact (0.999, with every parameter consistent); the others miss 1, 7, and 9 parameters. Codex with GPT-5.6 Terra earns 0.140 with lower geometry similarity (0.232) than five zero-reward attempts, because it stays within the failure budget. On this task, specification consistency rather than overall shape separates the rewarded attempts.

ModelAgentRewardggssFailedOutcome
Claude Fable 5.1Claude Code0.9990.9981.0000positive
Grok 4.6Grok Build0.6280.4820.9001positive
GPT-6 AstraCodex0.3750.5000.3007positive
GPT-5.6 TerraCodex0.1400.2320.1009positive
Claude Opus 5Claude Code0.0000.1460.00013specification zero
Claude Opus 5.5Claude Code0.0000.4450.00012specification zero
Claude Opus 5.5mini-swe-agent0.000———final exception
GPT-5.6 SolCodex0.0000.2990.00080specification zero
Gemini 3.8 FlashAntigravity0.0000.1950.00019specification zero
Gemini 3.8 Flashmini-swe-agent0.0000.2470.00017specification zero
Grok 4.7Grok Build0.0000.3190.00018specification zero
Kimi K3mini-swe-agent0.0000.2860.00019specification zero
Muse Spark 1.3mini-swe-agent0.0000.1260.00024specification zero

Table 7. Final outcomes of the 13 evaluated systems on the trussed-plate example freecad-2002df3ce7. gg and ss are the geometry and specification scores; Failed counts the parameters, of 86, that are inconsistent or not found. —: no scored output after a final exception.

The clevis connector (Figure 2) shows the opposite pattern (Table 8). Twelve of the 13 systems earn positive reward, and Claude Opus 5 with Claude Code earns 1.000 to three decimals, with every parameter consistent. Only Codex with GPT-5.6 Terra receives zero, on both axes. The other eleven positive outcomes miss at most 7 of the 44 parameters, but their geometry similarity ranges only from 0.175 to 0.500. Two of them, Claude Fable 5.1 with Claude Code and Gemini 3.8 Flash with mini-swe-agent, pass every parameter check yet earn 0.419 because their geometry similarity is 0.265. Here, geometry rather than the named dimensions limits most rewards.

ModelAgentRewardggssFailedOutcome
Claude Opus 5Claude Code1.0001.0001.0000positive
Claude Opus 5.5Claude Code0.6150.5000.8002positive
Claude Fable 5.1Claude Code0.4190.2651.0000positive
Gemini 3.8 Flashmini-swe-agent0.4190.2651.0000positive
GPT-6 AstraCodex0.3970.2640.8002positive
Kimi K3mini-swe-agent0.3970.2640.8002positive
Muse Spark 1.3mini-swe-agent0.3970.2640.8002positive
Grok 4.7Grok Build0.3070.1900.8002positive
Gemini 3.8 FlashAntigravity0.2990.1900.7003positive
Claude Opus 5.5mini-swe-agent0.2520.2180.3007positive
Grok 4.6Grok Build0.2520.2180.3007positive
GPT-5.6 SolCodex0.2430.1750.4006positive
GPT-5.6 TerraCodex0.0000.0000.00026geometry and specification zero

Table 8. Final outcomes of the 13 evaluated systems on the clevis-connector example freecad-fa0a1abef3. Columns as in Table 7; Failed counts the parameters, of 44, that are inconsistent or not found.

Discussion and Limitations

The benchmark is a curated 100-task FreeCAD evaluation, not a representative sample of all engineering CAD. Family dependencies and selection choices limit generalization. Native singlebody PartDesign requirements do not cover general assembly mating, FEM/FEA, manufacturing qualification, or broad organic-surface modeling.

Create-and-edit measures two-stage construction, not maintenance of an externally authored document. Grader rewards are heuristic proxies: geometric similarity and checked parameters do not certify all future edits, design intent, or engineering validity. Alternative valid constructions may challenge reference-based comparisons.

Public assets, providers’ undisclosed training data, and changing provider implementations limit contamination claims and reproducibility. Agent tools, prompts, serving paths, and budgets confound model-only comparisons. Single official attempts do not establish run-to-run robustness. Known API costs exclude some compute and hosting costs.

Conclusion

Parametric CAD Bench v3 evaluates agentic CAD authoring across text-based creation, create-and-edit, and engineering-drawing reconstruction. Its 100 tasks require native parametric FreeCAD models, evaluated through geometric similarity and CAD-engineer-curated specification checks.

Across 13 evaluated model–agent systems, the largest performance differences occur in image-to-CAD. Opus 5.5’s improvement over Opus 5 is concentrated in drawing reconstruction, while agent comparisons reveal workflow-dependent strengths rather than a universally superior interface. Create-and-edit remains challenging: five tasks receive zero reward from every evaluated system. Cost and speed also distinguish systems; Codex with GPT-6 Astra offers the lowest observed API cost per score point and the shortest mean trial duration.

These results highlight substantial progress alongside remaining limitations in reliable parametric CAD construction. By releasing versioned tasks, execution configurations, scoring code, and inspectable run artifacts, Parametric CAD Bench provides a versioned, inspectable foundation for measuring progress in engineering-oriented CAD agents.

Reproducibility and Artifact Index

The following public resources accompany this paper:

The frozen dataset digest is

sha256:8a786725d3648a76282fc5868df06da3d5e00d603f283573a53430803465d78a.

The reported runs use Harbor 0.23.0, FreeCAD 1.1.0, and gnucleus-freecad-validator 0.6.0, published on PyPI and tagged v0.6.0 (commit 2a5d67f) in the validator repository. The trials of the linked Harbor jobs include agent logs, generated models, and saved verifier outputs.

The benchmark tasks, including their specifications, reference models, and drawings, were authored by gNucleus AI and are released under the Apache License 2.0, as is the validator. The task images redistribute FreeCAD and Open CASCADE Technology under the LGPL-2.1, with their license notices.

Apart from the four audited API costs described in § 5.4, every reported statistic is computed from the public tasks and the final trials of the public Harbor jobs, and can be recomputed from them.

Text-to-CAD Creation Example Specifications

For each creation example, the agent receives the description and key parameters below, reproduced exactly as given, including their full decimal precision. The instruction adds the output contract shared by creation tasks: write a FreeCAD 1.1.0 script to /app/answer.py that saves /app/answer.FCStd, building the model as a parametric feature tree in a single PartDesign Body that produces exactly one solid, with editable features rather than a pre-computed shape.

Trussed upper plate (freecad-2002df3ce7)

The key parameters are listed in Table 9.

ParameterValueUnitCheck
Plate outline, thickness, and side recesses
plate_thickness8.00mmW
base_sketch_plane_rotation_angle90.0degN
large_outer_rectangle_width280.00mmN
ParameterValueUnitCheck
large_outer_rectangle_height447.00mmN
large_outer_rectangle_center_y−51.00mmN
middle_outer_rectangle_width144.00mmN
middle_outer_rectangle_height64.00mmN
middle_outer_rectangle_center_y204.50mmN
small_outer_rectangle_width52.00mmN
small_outer_rectangle_height38.00mmN
small_outer_rectangle_center_y255.50mmN
rectangular_section_corner_chamfer_fillet_radius10.00mmW
large_outer_rectangle_lower_corner_chamfer_flat_width200.00mmN
large_outer_rectangle_upper_corner_chamfer_setback17.00mmN
large_outer_rectangle_side_trapezoidal_recess_depth17.00mmN
large_outer_rectangle_side_trapezoidal_recess_01_center_y−149.50mmN
large_outer_rectangle_side_trapezoidal_recess_01_outer_height98.28mmN
large_outer_rectangle_side_trapezoidal_recess_01_inner_flat_height47.72mmN
large_outer_rectangle_side_trapezoidal_recess_02_center_y25.50mmN
large_outer_rectangle_side_trapezoidal_recess_02_outer_height148.28mmN
large_outer_rectangle_side_trapezoidal_recess_02_inner_flat_height97.72mmN
Truss layout and diagonal angles
truss_layout_symmetry_axis_x0.00mmN
truss_cutout_transition_radius3.00mmW
truss_diagonal_angle_family_127.3777028degN
truss_diagonal_angle_family_263.4349488degN
truss_diagonal_angle_family_328.8386275degN
truss_diagonal_angle_family_446.1233027degN
truss_diagonal_angle_family_5165.8183429degN
Mounting holes and center bore
mounting_hole_diameter4.30mmW
center_bore_diameter7.00mmW
center_bore_axis_y264.50mmN
center_mounting_hole_pair_half_spacing18.00mmN
upper_inner_mounting_hole_pair_half_spacing62.00mmN
lower_inner_mounting_hole_pair_half_spacing61.76mmN
upper_inner_mounting_hole_pair_y226.50mmN
lower_inner_mounting_hole_pair_y165.50mmN
side_mounting_hole_axis_x136.00mmN
side_mounting_hole_row_1_y137.50mmN
side_mounting_hole_row_2_y113.50mmN
side_mounting_hole_row_3_y−62.50mmN
side_mounting_hole_row_4_y−86.50mmN
side_mounting_hole_row_5_y−216.32mmN
side_mounting_hole_row_6_y−240.32mmN
Triangular and central openings
large_triangular_opening_pair_01_02_base_length141.90mmN
large_triangular_opening_pair_01_02_base_angle27.3777028degN
large_triangular_opening_pair_01_positive_member_centroid_x86.75mmN
large_triangular_opening_pair_01_positive_member_centroid_y−170.50mmN
large_triangular_opening_pair_02_positive_member_centroid_x24.25mmN
large_triangular_opening_pair_02_positive_member_centroid_y−170.50mmN
large_triangular_opening_pair_03_positive_member_centroid_x86.78mmN
large_triangular_opening_pair_03_positive_member_centroid_y16.50mmN
large_triangular_opening_pair_03_base_length133.12mmN
ParameterValueUnitCheck
large_triangular_opening_pair_03_base_angle28.8386275degN
medium_triangular_opening_pair_01_02_base_length73.49mmN
medium_triangular_opening_pair_01_02_base_angle62.6222972degN
medium_triangular_opening_pair_01_positive_member_centroid_x55.50mmN
medium_triangular_opening_pair_01_positive_member_centroid_y−230.85mmN
medium_triangular_opening_pair_02_positive_member_centroid_x55.50mmN
medium_triangular_opening_pair_02_positive_member_centroid_y−110.15mmN
medium_triangular_opening_pair_03_center_x30.11mmN
medium_triangular_opening_pair_03_center_y−7.95mmN
medium_triangular_opening_pair_03_envelope_width36.23mmN
medium_triangular_opening_pair_03_envelope_height66.89mmN
right_triangular_opening_pair_base_length73.30mmN
right_triangular_opening_pair_base_angle61.1613725degN
right_triangular_opening_01_centroid_x55.50mmN
right_triangular_opening_01_centroid_y−40.31mmN
central_trapezoidal_opening_center_y96.50mmN
central_trapezoidal_opening_corner_radius10.00mmW
central_trapezoidal_opening_envelope_width184.30mmW
central_trapezoidal_opening_envelope_height118.00mmW
central_trapezoidal_opening_lower_flat_width108.61mmN
small_triangular_opening_centroid_x0.00mmN
small_triangular_opening_centroid_y186.39mmN
small_triangular_opening_base_length86.69mmN
small_triangular_opening_base_angle43.8766972degN
chamfer_cut_right_triangular_opening_01_center_x−32.55mmN
chamfer_cut_right_triangular_opening_01_center_y203.69mmN
chamfer_cut_right_triangular_opening_01_envelope_width38.90mmN
chamfer_cut_right_triangular_opening_01_envelope_height37.63mmN
chamfer_cut_right_triangular_opening_02_center_x32.55mmN
chamfer_cut_right_triangular_opening_02_center_y203.69mmN
chamfer_cut_right_triangular_opening_02_envelope_width38.90mmN
chamfer_cut_right_triangular_opening_02_envelope_height37.63mmN
Second right-triangular opening
right_triangular_opening_02_centroid_x−55.50mmN
right_triangular_opening_02_centroid_y−40.31mmN

Table 9. Key parameters of freecad-2002df3ce7, as given to the agent. Check: W, a witness on the final geometry; C, a custom binding; N, a native sketch or feature measurement by the task’s param_check.py.

A trussed upper plate consists of a thin, planar outer frame with rounded corners, repeated diagonal truss webs, open cutouts, peripheral mounting holes, and a centered end extension.

The outer profile is centered on truss_layout_symmetry_axis_x and is the union of three rectangular sections: large_outer_rectangle, middle_outer_rectangle, and small_outer_rectangle. The large rectangle has two mirrored trapezoidal side recesses on each side, with equal diagonal flanks, and a centered lower flat bounded by symmetric rounded chamfers. The two upper corners of the large rectangle have chamfer setback large_outer_rectangle_upper_corner_chamfer_setback. All rectangular-section corner chamfers use rectangular_section_corner_chamfer_fillet_radius.

The truss layout is centered on truss_layout_symmetry_axis_x. All named layout x and y coordinates use a sketch frame at the global origin. Rotate this frame from the global XY plane by -base_sketch_plane_rotation_angle about global +X, using the right-hand convention. Thus layout +x points along global +X, layout +y points along global -Z, and the plate extends from global Y=0 in the +Y direction by plate_thickness. Its triangular substructures are six large triangles, six medium triangles, two right triangles, one small triangle, and two chamfer-cut right triangles. Each strictly isosceles triangle is defined by its centroid, ideal base length, and base angle; its rounded profile uses truss_cutout_transition_radius. large_triangular_opening_pair_01 through _03 and medium_triangular_opening_pair_01 through _02 each define two mirrored isosceles openings. The first and third large pairs have exterior vertical bases, while the second has interior vertical bases. The first medium pair has lower horizontal bases and the second has upper horizontal bases. medium_triangular_opening_pair_03 has an interior vertical base and two diagonal flanks meeting at a rounded exterior apex. An upper horizontal truncation joins the interior base to the upper flank; all four transitions are rounded with truss_cutout_transition_radius. The two right triangles and the small triangle have lower horizontal bases and upward apices. The chamfer-cut right triangles have upper horizontal bases, exterior vertical legs, and rounded diagonal cut corners.

central_trapezoidal_opening defines the remaining non-triangular internal loop; together with the seventeen triangular openings, it forms eighteen internal loops. It shares truss_layout_symmetry_axis_x with the outer profile and center bore. Its rounded six-sided boundary has an upper horizontal edge, two upper vertical sidewalls, two lower flanks sloping inward toward the bottom, and a lower horizontal edge. The lower flanks are inclined by truss_diagonal_angle_family_3 from the vertical. The finished straight lower edge has central_trapezoidal_opening_lower_flat_width. All six corners are tangent rounds of central_trapezoidal_opening_corner_radius; the named envelope and center locate the complete boundary. The plate body is governed by plate_thickness. The triangular transitions use truss_cutout_transition_radius, while central_trapezoidal_opening uses central_trapezoidal_opening_corner_radius. The mounting holes and center bore are governed by their respective diameters, pair spacings, rows, and axes.

Asymmetric clevis joint connector (freecad-fa0a1abef3)

The key parameters are listed in Table 10.

ParameterValueUnitCheck
Central body
connector_body_length75.0mmN
connector_body_width45.0mmN
connector_body_thickness12.0mmW
Large split-ring clevis
clevis_end_outer_diameter45.0mmW
pin_hole_diameter25.0mmW
clevis_slot_width_gap5.0mmW
clevis_arc_center_x_offset37.5mmN
clevis_body_depth50.0mmW
clevis_body_depth_offset−20.0mmN
first_clevis_outer_slot_tangent_angle6.3793702degN
first_clevis_inner_slot_tangent_angle11.536959degN
first_clevis_inner_prong_tangent_x49.7474487mmN
first_clevis_outer_prong_tangent_x59.8606798mmN
Clevis pin-hole cut
pin_hole_depth34.5mmN
pin_hole_center_offset_x37.5mmN
Fork prongs
clevis_fork_outer_arc_radius17.5mmW
clevis_fork_inner_arc_radius7.5mmN
clevis_fork_arc_central_angle180degN
clevis_fork_prong_straight_length15.0mmN
clevis_fork_prong_thickness8.0mmN
Fork U-shaped opening
clevis_slot_arc_radius7.5mmN
clevis_slot_arc_central_angle180.0degN
clevis_slot_straight_length15.0mmN
clevis_slot_profile_width15.0mmN
clevis_slot_depth34.5mmN
Side arm
clevis_pin_hole_diameter15.0mmN
clevis_end_arc_radius20.0mmN
ParameterValueUnitCheck
clevis_end_arc_central_angle180.0degN
clevis_arm_length46.7196386mmN
clevis_arm_width40.0mmN
clevis_arm_depth45.0mmN
clevis_arm_profile_start_x52.9513389mmN
clevis_arm_end_center_x99.6709775mmN
clevis_arm_end_center_z_offset9.0mmN
Central relief slot
clevis_slot_length74.88mmN
clevis_slot_width5.21mmW
clevis_slot_y_min−2.71mmN
Upper and lower relief slots
upper_slot_length72.25mmN
upper_slot_width6.96mmN
upper_slot_y_min16.3556146mmN
lower_slot_length73.8mmN
lower_slot_width10.06mmN
lower_slot_y_min−26.4156146mmN
slot_depth73.6mmN

Table 10. Key parameters of freecad-fa0a1abef3, as given to the agent. Check: W, a witness on the final geometry; C, a custom binding; N, a native sketch or feature measurement by the task’s param_check.py.

An asymmetric clevis joint connector has a centered rectangular body, a large split-ring clevis at its positive-X end, a smaller fork at the negative-X end, and a long rounded side arm extending beyond the large clevis. Use global X along the connector, global Y across its width, and global Z through its thickness; the two ends are not mirror images.

The central body uses the listed connector_body dimensions. The large clevis is centered at clevis_arc_center_x_offset, opens toward positive X, and uses clevis_end_outer_diameter, pin_hole_diameter, clevis_slot_width_gap, and the first-clevis tangent controls for its split-ring profile. Its inner and outer circular boundaries share a center, and the pin-hole cut enters from the positive-Z face. Its Z placement is set by clevis_body_depth and clevis_body_depth_offset, and its pin-hole cut uses pin_hole_center_offset_x and pin_hole_depth. The clevis_body_depth_offset is measured from the central body’s positive-Z face. The smaller fork opens toward negative X, with its rounded center one clevis_fork_prong_straight_length inward from the body end. Its concentric prong arcs, U-shaped opening, and positive-Z projection use the corresponding clevis_fork and clevis_slot parameters. The U-shaped opening is concentric with the rounded fork end and is cut from the positive-Z face toward negative Z. clevis_fork_prong_thickness is the extrusion thickness rather than the radial wall thickness. This fork has no separate pin hole.

The long side arm lies against the front Y face and runs from clevis_arm_profile_start_x to the rounded end located by clevis_arm_end_center_x and clevis_arm_end_center_z_offset. Its profile uses the listed clevis_arm and clevis_end dimensions, and clevis_pin_hole_diameter defines a concentric hole through its rounded end parallel to global Y. The central, upper, and lower relief slots all belong to this arm: they begin at clevis_arm_profile_start_x, extend toward and open through the rounded end, and use their listed lengths, widths, transverse positions, and common slot_depth. All three relief slots enter from the positive-Z face and cut toward negative Z.

Text-to-CAD Create-and-Edit Example Specification

For freecad-ff7883ab0c, the following base description, edit instruction, and parameters reproduce the supplied task specification. The shared output contract requires a FreeCAD 1.1.0 script at /app/answer.py and two documents derived from the script filename at runtime: answer_base.FCStd and answer_edit.FCStd. Each must contain an editable parametric feature tree in a single PartDesign Body producing one solid.

Base part. An axisymmetric flanged inlet housing has a continuous central bore, a tapered inlet rim, a tubular middle body, a tapered rear hub, and a plain cylindrical rear flange disk. Along the axis, the rear hub widens into the flange disk, the body narrows through a short taper and continues cylindrically, and the inlet end expands through its tapered rim. No contoured front plate or rear-face circular opening pattern is present yet.

Edit instruction. Add the front flange plate and rear opening pattern to the axisymmetric inlet body. Starting at tapered_inlet_rim_axial_end_position, extend the fourfold-symmetric front plate forward by square_flange_thickness while preserving the existing central bore. Form its closed outline from the named side, corner, and transition arcs joined by the short tangent connectors. Add number_of_bolt_holes equal through holes from the signed seed center and distribute them uniformly around the housing axis. From the rear cylindrical flange face, cut number_of_flange_base_counterbores equal circular openings with flange_base_counterbore_diameter and flange_base_counterbore_depth, again using the signed seed center and uniform angular spacing. Keep the tubular body, inlet rim, rear hub, rear disk envelope, and central passage unchanged.

Table 11: Text-to-CAD create-and-edit example parameters, reproduced from the agent instruction. The 15 base parameters remain unchanged; the edit adds the 14 target requirements below.

ParameterValueUnit
Base part (unchanged by the edit)
central_bore_diameter50.8mm
tubular_body_outer_diameter68.58mm
tubular_body_taper_end_diameter75.2mm
tapered_inlet_rim_small_diameter88.84mm
tapered_inlet_rim_large_diameter95.25mm
tapered_hub_small_diameter92.08mm
tapered_hub_large_diameter97.78mm
rounded_flange_outer_diameter152.4mm
tapered_hub_axial_start_position−27.36mm
rounded_flange_axial_start_position−24.51mm
tubular_body_axial_start_position−8.0mm
tubular_body_axial_middle_position7.57mm
tubular_body_axial_end_position26.8mm
tapered_inlet_rim_axial_start_position26.8mm
tapered_inlet_rim_axial_end_position30.0mm
Additional target requirements
square_flange_thickness16.51mm
square_flange_side_arc_radius63.5mm
square_flange_corner_arc_radius19.4447813mm
square_flange_transition_arc_radius25.4mm
square_flange_corner_arc_center_offset40.132mm
number_of_bolt_holes4
bolt_hole_diameter13.34mm
bolt_hole_seed_center_x_offset42.6562166mm
bolt_hole_seed_center_y_offset42.6562166mm
flange_base_counterbore_depth34.54mm
number_of_flange_base_counterbores4
flange_base_counterbore_diameter19.05mm
flange_base_counterbore_seed_center_x_offset−42.72mm
flange_base_counterbore_seed_center_y_offset−42.83mm

Table 11. Text-to-CAD create-and-edit example parameters, reproduced from the agent instruction. The 15 base parameters remain unchanged; the edit adds the 14 target requirements below.

Supplementary Reference-Model Statistics

Figure 6. Feature-object types across the 100 final references, broken down by workflow. A task can contribute several objects of the same type.

Figure 7. Feature counts for the 30 text-to-CAD create-and-edit pairs. Left: side-by-side bars distinguish base and edited target distributions. Right: paired base/target counts; circle area and numerals show how many tasks share each pair. The dashed diagonal indicates no count change. The target-minus-base count is not necessarily the number of operations needed to perform an edit.

Parametric CAD Bench v3: reference-model statistics. Four histograms: All tasks (one final reference each) (n=100), mean=6.42; Text-to-CAD Creation (n=30), mean=4.87; Text-to-CAD Create-and-Edit (target) (n=30), mean=5.03; Image-to-CAD (n=40), mean=8.62. Axes: Sketch objects and Number of tasks.

Figure 8. Saved sketch-object counts per final reference model. Sketch objects are excluded from the solid-modeling feature statistic.

Leaderboard Figures

Each chart occupies a landscape page to keep all 13 model–agent labels readable.

Parametric CAD Bench v3; Text-to-CAD Creation | 30 tasks; bar chart of mean continuous task reward (%) for 13 model–agent systems

Figure 10. Text-to-CAD creation mean reward across 30 tasks. Systems are ordered by their creation point estimates; workflow-specific intervals are not plotted.

Parametric CAD Bench v3 Text-to-CAD Create-and-Edit benchmark bar chart

Figure 11. Text-to-CAD create-and-edit mean reward across 30 tasks. Each task scores both base and target artifacts through the nested harmonic aggregation in Equation (5). Bars show point estimates, not binary success rates.

References

  1. [1]BenchCAD. BenchCAD 2.0: Benchmarking programmatic CAD on real engineering work. Blog post, October 2026. URL https://benchcad.com/blog/benchcad-2.html. Published October 12, 2026; accessed November 1, 2026.
  2. [2]BenchCAD. BenchCAD: A benchmark for programmatic CAD, 2026. URL https://benchcad.com/. Accessed October 6, 2026; includes agentic evaluation results.
  3. [3]CADBench Contributors. CADBench: Evaluation code and submission requirements. Official project repository, 2026. URL https://github.com/anniedoris/CADBench. Accessed October 6, 2026.
  4. [4]CADGenBench Contributors. CADGenBench benchmark submission requirements, 2026. URL https://github.com/huggingface/cadgenbench/blob/main/docs/benchmark/submission.md. Accessed October 6, 2026.
  5. [5]CADGenBench Contributors. CADGenBench validation policy, 2026. URL https://github.com/huggingface/cadgenbench/blob/main/docs/benchmark/validation.md. Accessed October 6, 2026.
  6. [6]CadQuery Contributors. CadQuery documentation: Introduction. Official documentation, n.d. URL https://cadquery.readthedocs.io/en/latest/intro.html. Accessed October 6, 2026.
  7. [7]Anna C. Doris, Jacob Thomas Sony, Ghadi Nehme, Era Syla, Amin Heyrani Nobari, and Faez Ahmed. CADBench: A multimodal benchmark for AI-assisted CAD program generation, 2026. URL https://arxiv.org/abs/2605.10873. Version 2, June 18, 2026.
  8. [8]FreeCAD Contributors. FreeCAD: Your own 3d parametric modeler. Official project repository and overview, n.d. URL https://github.com/FreeCAD/FreeCAD. Accessed October 6, 2026.
  9. [9]Yipeng Gao, Lei Shu, Genzhi Ye, Xi Xiong, Ameesh Makadia, Meiqi Guo, Laurent Itti, and Jindong Chen. 3DCodeBench: Benchmarking agentic procedural 3d modeling via code, 2026. URL https://arxiv.org/abs/2606.01057.
  10. [10]gNucleus AI. FreeCAD Validator 0.6.0: Geometry and specification scoring implementation. Versioned source code, 2026. URL https://github.com/gNucleus-AI/freecad-validator/tree/v0.6.0/src/freecad_validator. Version 0.6.0, default v2 scorer.
  11. [11]gNucleus AI. Parametric CAD Bench: Can ai agents author editable cad? Benchmark release post, May 2026. URL https://cadbench.ai/news/parametric-cad-bench. Published May 13, 2026.
  12. [12]gNucleus AI. CAD Bench v2: A corrected benchmark for the new frontier. Benchmark release post, September 2026. URL https://cadbench.ai/news/cad-bench-v2. Published September 8, 2026.
  13. [13]gNucleus AI. Parametric CAD Bench V3: Text, editing, and engineering drawings. Benchmark release post, September 2026. URL https://cadbench.ai/news/cad-bench-v3. Published September 21, 2026.
  14. [14]gNucleus AI. Parametric CAD Bench V3: Claude Opus 5.5 takes the lead. Leaderboard update post, September 2026. URL https://cadbench.ai/news/cad-bench-v3-opus-5-5. Published September 24, 2026.
  15. [15]gNucleus AI. FreeCAD tasks contributed to Terminal-Bench. Task implementations in the Terminal-Bench repository, 2026. URL https://github.com/harbor-framework/terminal-bench/tree/main/tasks. Tasks: freecad-impeller, freecad-spring-clip, and freecad-platform-drawing; accessed October 6, 2026.
  16. [16]Harbor Framework. Harbor. Framework website, 2026. URL https://harborframework.com. Accessed October 6, 2026.
  17. [17]Mike A. Merrill et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. URL https://arxiv.org/abs/2601.11868.
  18. [18]Open CASCADE. Open CASCADE Technology: Modeling data. Official user guide, n.d. URL https://dev.opencascade.org/doc/overview/html/occt_user_guides__modeling_data.html. Accessed October 6, 2026.
  19. [19]Michael Rabinovich and Hugging Face. CADGenBench: A benchmark for AI-driven CAD generation. Benchmark and leaderboard website, 2026. URL https://huggingface.co/spaces/HuggingAI4Engineering/CADGenBench. Accessed October 6, 2026.
  20. [20]Dhruv Saini. CAD-bench: Benchmarking language models on functional CAD generation. Poster, The Second Workshop on Agents in the Wild: Safety, Security, and Beyond, ICML 2026, 2026. URL https://icml.cc/virtual/2026/67851. Accessed November 1, 2026.
  21. [21]Haozhe Zhang, Kaichen Liu, Miaomiao Chen, Lei Li, Shaojie Yang, Cheng Peng, and Han jie Chen. BenchCAD: A comprehensive, industry-standard benchmark for programmatic CAD, 2026. URL https://arxiv.org/abs/2605.10865.

Paper details

vlnc:W_99jxxtnk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents