WorksComputer Science
Beyond Source Code: Evaluating Natural Language Specifications for LLM-Based C-to-Rust Translation
Abstract
Large language models (LLMs) have shown promising results for C-to-Rust translation, yet existing approaches primarily rely on source code as input, leaving natural-language specifications of program behavior underexplored as an alternative or complementary representation. We hypothesize that direct code translation encourages LLMs to preserve C-specific implementation choices, producing Rust that may be functionally correct but unnecessarily verbose, reliant on unsafe constructs, and less idiomatic. We investigate this hypothesis by comparing three translation settings: code-only, specification-only, and code-and-specification. Using 300 C programs from the IBM CodeNet dataset, we evaluate GPT-5.2 and GPT-4o-mini on functional correctness, maintainability, complexity, idiomaticity, unsafe construct usage, and cost. Code-only translation achieves the highest test pass rate (98.6%) but often mirrors C's implementation patterns. In contrast, specification-only translation produces the most idiomatic and compact Rust, reducing unsafe blocks by approximately 6x and Clippy style warnings by 2.4x. However, using specifications without validation yields test pass rates as low as 86% with our best performing model. Adding an iterative test-based validation and refinement step for specifications raises this to a 95.3% test pass rate. Overall, specification-based translation consistently improves code quality relative to code-only translation, while code-and-specification offers the best balance between correctness (98.3% test pass rate) and maintainability. Our study systematically compares these translation settings, highlighting the trade-offs among specification quality, translation correctness, code quality, and computational cost.
Introduction
Software written in C underpins critical systems such as operating systems, embedded firmware, and network infrastructure [1]. Although C offers high performance and fine-grained control over memory, this flexibility comes at the cost of memory safety. Buffer overflows, use-after-free errors, and null-pointer dereferences remain a major source of software vulnerabilities [2]. Microsoft and Google report that approximately 70% of their critical security bugs originate from memory-unsafe languages such as C [3, 4]. Consequently, there is growing industrial interest in replacing legacy C with memory-safe alternatives. Rust has emerged as one of the most promising candidates for this migration. It provides performance comparable to C while enforcing memory safety at compile time through its ownership and borrowing system [5]. This has motivated large-scale migration initiatives in both industry and government [6, 7, 8]. Unfortunately, manually rewriting mature C systems in Rust is prohibitively expensive and error-prone. Existing transpilers, such as C2Rust [9], automate much of the syntactic conversion but largely preserve the original implementation, often producing Rust that resembles “C written in Rust syntax”, with excessive unsafe blocks, non-idiomatic constructs, and limited maintainability [10, 11]. As a result, these tools only partially realize the benefits that motivate migration to Rust in the first place.
Large Language Models (LLMs) have recently emerged as a promising alternative. Rather than performing mechanical translation, LLMs can synthesize Rust that is more natural, idiomatic, and easier to maintain [12, 13, 14]. Recent studies have demonstrated encouraging functional correctness for C-to-Rust translation [13, 15, 16, 14]. However, nearly all existing approaches adopt the same translation paradigm: the LLM receives the C source code and is asked to generate the corresponding Rust implementation. While effective for preserving program behavior, this code-only setting also anchors the model to the original implementation. Consequently, the generated Rust often inherits C-specific implementation choices, including low-level control flow, data representations, and programming patterns, rather than taking advantage of Rust’s higher-level abstractions and safety mechanisms.
This observation raises a question: can we provide the LLM with a representation that emphasizes the program’s intended behavior rather than its implementation? In this work, we investigate Natural Language Specifications [17], which we define as structured natural-language descriptions of program behavior, including inputs, outputs, preconditions, postconditions, invariants, error handling, functional description, state transitions and non-functional constraints. These specifications are generated from the original C source code, with one specification per program. Unlike formal specifications, they are expressed in natural language and do not provide mathematical guarantees of correctness or completeness. By emphasizing what a program does rather than how it is implemented, they may allow LLMs to produce more idiomatic Rust without being constrained by C-specific implementation details.
Recent work has begun exploring intermediate representations for LLM-based code translation. Dipto et al. [18], for example, generate step-by-step natural-language descriptions before translation, while Nitin et al. [17] use multi-modal specifications together with the source code. These studies demonstrate that richer intermediate representations can improve translation quality, but important questions remain unanswered. Existing natural-language intermediates primarily describe how a program operates rather than what behavior it guarantees. Likewise, natural language specifications have only been evaluated alongside the original source code, specifications that fail consistency checks are typically discarded rather than improved, and evaluations focus almost exclusively on functional correctness. As a result, it remains unclear whether test-validated specifications can serve as an effective standalone intermediate representation that improves not only correctness, but also maintainability, idiomaticity, and safety.
To answer these questions, we conduct a systematic comparison of three translation settings: code-only, specification-only, and code-and-specification. We also study the effect of adding an iterative test-based validation and refinement step for specifications. Given a specification, we use an LLM to regenerate a C program and check its behavior against the available test cases. If the regenerated program fails, we use the observed failures to refine the specification. We call a specification test-validated when the regenerated program passes the available tests; this provides evidence of agreement on the tested behavior, rather than a guarantee of complete semantic equivalence. Using 300 C programs from IBM CodeNet [17], we evaluate GPT-5.2 and GPT-4o-mini across all three settings, measuring not only functional correctness but also maintainability, complexity, idiomaticity, unsafe construct usage, and translation cost.
Our results reveal a clear trade-off between correctness and code quality. Code-only translation achieves the highest test pass rate (98.6%) but often mirrors C’s implementation choices. In contrast, specification-only translation produces the most idiomatic Rust, reducing Clippy style warnings by 2.4× and unsafe blocks by approximately 6×, while improving maintainability and reducing code complexity. Its main limitation is functional correctness: using specifications without validation yields test pass rates between 69% and 86%, depending on the experimental configuration. With test-based validation and refinement, the GPT-5.2 setting achieves a 95.3% test pass rate. Finally, combining source code with test-validated specifications process provides the best overall balance, achieving a 98.3% test pass rate while improving measured code quality relative to code-only translation.
Our main contributions are twofold: (i) a systematic comparison of code-only, specification-only, and code-and-specification translation, evaluating not only functional correctness but also maintainability, complexity, idiomaticity, unsafe construct usage, and cost; and (ii) an empirical evaluation of how test-based specification validation and refinement affect translation correctness and code quality, revealing trade-offs among specification quality, translation fidelity, code quality, and computational cost.
Related Work
Repository-Level Decomposition and Metadata-Driven Translation. A common strategy is to decompose C programs into functional units and translate them incrementally using structural information such as function signatures, type definitions, and call graphs [19, 15]. Feniser et al. [14] take a different angle, focusing on functional correctness through differential fuzzing and structured prompting, though their success rates range from 20–60% and drop sharply for programs over 100 lines. Tadesse et al. [11] compare three transpilers against human-written Rust across 18 quality categories, finding that reducing unsafe constructs often introduces other problems like runtime panics or redundant code, a trade-off that correctness-only evaluation simply does not capture. None of these approaches use natural language specifications or isolate a specification-only setting; they remain anchored to syntactic and structural artifacts.
Specifications and Program Properties as Translation Guides. Shetty et al. [20] extract program properties such as nullability, aliasing, and type information through dynamic analysis and use them to guide translation. These properties are used as-is, with no check that they are correct or complete, and they cover only a narrow slice of what a full specification can express. Nitin et al. [17] take a broader view, showing that static behavioral specifications, input-output examples, and natural language descriptions each contribute something different, and that combining them with source code improves translation effectiveness. However, three important questions remain unanswered by both studies. First, neither ever tests specifications on their own without source code alongside them. Second, when a specification fails a consistency check, Nitin et al. [17] discard it, and Shetty et al. [20] do not check at all. Neither tries to fix a specification that turns out to be wrong. Third, both studies measure only whether translations pass test cases, with no attention to whether the resulting code is maintainable, idiomatic, or safe. We address these three limitations in our work.
Intermediate Representations and Prompt-Centric Refinement. A third line of work asks whether giving the model a richer intermediate step can improve translation. Tai et al. [21] have the model first summarize the source program in natural language before translating, gaining up to 13.8% accuracy improvement on CodeNet and AVATAR [22]. Dipto et al. [18] take a similar approach with a structured step-by-step description of control flow and I/O, improving accuracy by 10.8%. Macedo et al. [23] find that between 26.4% and 73.7% of translations require post-processing because model outputs mix code with natural language, while Ye et al. [24] improve results through iterative prompt refinement, achieving gains of up to 13.68% on Java-to-Python translation. The intermediates used by Tai et al. [21] and Dipto et al. [18] describe how a program works rather than what it guarantees, and neither is checked against the source program before being used. The prompt-centric approaches tend to be problem-specific and do not generalize well across diverse inputs. Rather than refining the prompt, we refine the intermediate artifact itself, improving the specification through test-based validation until it accurately captures the program’s behavior.
Study Design
The goal of this study is to investigate whether natural language specifications can serve as an effective intermediate representation for LLM-based C-to-Rust translation. Specifically, we examine whether reasoning from a program’s intended behavior, rather than solely from its implementation, leads to translations that are not only functionally correct but also more maintainable, idiomatic, and safe. Our study is guided by the following research questions (RQs):
RQ1: How does test-based validation and refinement of specifications affect specification-based translation? Specification-based translation can only be as correct as the specification it starts from. This RQ compares unvalidated and test-validated specifications to measure how much checking and refining a specification before translation changes translation correctness. We assess whether it carries enough information to reproduce the program’s behavior.
RQ2: What is the impact of specifications on the trade-off between translation correctness and code quality? Code-only translation gives the LLM access to the implementation, so we expect it to achieve the highest correctness. This RQ asks whether that correctness advantage comes at the cost of code quality, and whether specification-based translation can close that quality gap without sacrificing too much correctness.
RQ3: To what extent do specifications encourage implementation choices that are less constrained by the original source implementation? This RQ looks past aggregate metrics to ask what is actually happening at the code level: when an LLM translates from a specification instead of code, does it produce implementations that are free from C’s structural choices, or does it still end up mirroring C’s structure?
RQ4: What are the cost–quality trade-offs of specification-based translation pipelines? Generating, validating, and refining specifications all require additional LLM calls beyond translation itself. This RQ asks whether the quality gains from specification-based translation justify their added cost.
Figure 1 provides a high-level overview of our study. Given a source program, we first generate a natural-language specification using an LLM, then translate the source program into the target language across three settings (i.e., code-only, specification-only, and code-and-specification). These three settings allow us to systematically isolate and measure the contribution of specifications to translation quality. In the following subsections, we detail our dataset selection criteria, LLM configuration, and specification generation strategy.
Figure 1. Overview of Translation Approach.
Dataset
We leverage the IBM CodeNet [25] dataset as the primary corpus for this study. CodeNet comprises over 4,000 programming problems and test cases, in multiple programming languages, collected from competitive programming platforms. This dataset has been adopted in several studies [24], [23], [26], [27]. To ensure compatibility with prior work, we adopt the subset of 300 function-level code submissions written in C sampled by Nitin et al. [17]. Thus, our study focuses exclusively on the C-to-Rust translation setting. This choice is also motivated by the growing interest in migrating legacy C systems to Rust [28], [10].
LLM Selection
We select two LLMs from OpenAI, namely GPT-5.2 (latest model as the time of conducting this experiment) and GPT-4o-mini due to the wide adoption of these family of models in recent works [17], [12]. These models were chosen to balance high-capability reasoning models with more lightweight, cost-efficient alternatives, enabling a comprehensive evaluation of different performance and efficiency trade-offs. We set the temperature to 0 to ensure deterministic output, while keeping all other model parameters at their defaults.
Specification Generation, Validation, and Refinement
To derive natural language specifications, we prompt an LLM to generate structured specifications directly from the given code. Our specification format builds on the specifications used by Shetty et al. [20] and Nitin et al. [17], extending their structure with additional sections, namely invariants, error handling, state transitions, and non-functional constraints, and with explicit instructions asking the LLM to separate what a program guarantees from how it achieves it. The specification describes the observable behavior of a self-contained program. It does not describe memory-level properties such as ownership, aliasing, or resource lifetimes, nor does it capture cross-function contracts, shared global state, or concurrency. These properties are central to repository-scale C-to-Rust migration but largely absent from the self-contained CodeNet programs we study; extending the format to express them is left to future work. Listing 1 presents the prompt template for the specification generation.
Using this prompt, we consider two scenarios. In the first scenario, we use the LLM to generate the specification of a given C code and use this specification for translation to Rust in future steps without validation to check whether the specification is correct or not. In the second scenario, we incorporate a test-based validation step to assess the correctness of the generated specification before we utilize it for translation. Concretely, after generating the specification using the same prompt in Listing 1, we prompt the LLM in a separate request to synthesize an equivalent C program from the generated specification. The resulting program is then executed against the original test inputs.
If the synthesized program passes all test cases, we consider the specification test-validated: it contains enough information to reproduce the program’s behavior on the available tests. This does not establish that the specification is complete or correct beyond the tested behaviors; incorrect or extraneous statements that the tests do not exercise may remain undetected. If the synthesized program fails the test cases, we apply a specification diagnostic step consisting of two prompts, which we have made available in our supplementary material [29]. In this step, we prompt the LLM to identify discrepancies between the generated specification and the original C program, and between expected and actual outputs. The resulting diagnosis is then used, together with the original source code and the current specification, to refine the specification using a subsequent LLM prompt. The test cases used for validation are the same ones later used to evaluate the Rust translations. To limit how directly test information enters the specification, the refinement prompt does not receive the test cases themselves, only the natural-language diagnosis; the specification is thus revised based on a description of the behavioral gap rather than on specific input–output pairs. This process is repeated until the program regenerated from the specification passes all available tests. If, after attempts ( in our study), the synthesized C program still fails, we retain the most recently generated specification for that problem as the final output. This process is shown in Figure 1. Throughout the paper, we call specifications produced by this process test-validated, and specifications used directly after generation, without this check, unvalidated. For those where the regenerated program still failed after attempts; we retain the last specification for these programs and include them in the test-validated set.
Translation to Rust
We evaluate C-to-Rust translation under three variants: code-only, specification-only, and code-and-specification. Each variant allows up to repair attempts, a limit we chose based on observed diminishing returns in success rates beyond this point. When an initial translation fails, the LLM is iteratively prompted to repair it using an execution report derived from the previous attempt. This report may include Compilation Errors (CE), Runtime Errors (RE), Timeout Failures (TO), or mismatches between expected and actual outputs (test failures). Repair continues until a correct translation is produced or all attempts are exhausted. The prompts used at this stage are provided in our supplementary material [29].
You are an expert senior software engineer.
Given source code, generate a precise, structured
behavioral specification that states WHAT the code
guarantees, not HOW it achieves it.
Output must use exactly these sections and headings:
1. Functional Description
2. Inputs
3. Outputs
4. Preconditions
5. Postconditions
6. Invariants
7. Error Handling / Exceptional Behavior
8. State Transitions (if applicable)
9. Non-functional Constraints (if inferable)
Content requirements:
- Be concise and contract-focused; omit incidental or
low-level details.
- Do not reference language/library APIs, format strings,
or identifiers (e.g., no mentions of scanf, printf,
fgets, variable or function names, loop types, or
recursion).
- Describe I/O abstractly (e.g., "reads values from
standard input" or "returns a value"); never cite exact
parsing patterns or format specifiers.
- Do not restate implementation structure or control
flow. Focus on observable behavior, data constraints, and
guarantees.
- If behavior is ambiguous, state explicit assumptions.
If something cannot be deduced, write "Not inferable from
code."
- Avoid extreme assumptions. Capture only behavior
implied by the code’s observable contract.
- If the program is an executable, treat printing/writing
as side effects in Outputs.
- No code, no pseudocode, no preamble or epilogue text.
Use only the nine sections above.
- Return only the nine sections with the exact headings
and numbering specified. Do not include any other text,
examples, or code.
Now generate the specification for the code below:
<code>
Listing 1: Prompt template used for specification generation
Internal Code Quality Measurements
To measure the internal quality of the translated Rust code, we use rust-code-analysis [30], a static analysis tool developed by Mozilla for computing source code metrics, which has also been adopted in prior work on Rust code quality evaluation [31]. We use it to compute maintainability (Maintainability Index, SEI variant), complexity (Cyclomatic and Cognitive Complexity), and size and effort metrics (Halstead Volume and Effort, SLOC, PLOC, LLOC, comment density, and blank lines).
Idiomaticity Measurement
To measure the idiomaticity of the generated Rust code, we use Clippy [32], the official Rust linter. We exclude lints from the nursery and restriction categories, as nursery lints are still under development and may produce false positives, and Clippy’s own documentation advises against enabling restriction lints by default due to their subjective nature. We also exclude pedantic lints since they are disabled by default, are opinionated, and may produce false positives. Finally, we exclude the cargo category as those lints target Cargo.toml configuration files rather than the source code itself.
We normalize raw warning counts by the original C lines of code (LOC), putting all three settings on a common baseline. This allows comparing how many issues each setting introduces per unit of input program, a consistent approach used in previous papers [31].
In addition to idiomaticity, we also measure the use of unsafe constructs in the generated Rust code. A key motivation for migrating C code to Rust is to leverage Rust’s safety guarantees, so we want to know whether the translated code actually avoids unsafe constructs or simply replicates C-style patterns. We measure three things: the number of unsafe blocks, the number of lines covered by unsafe blocks, and the number of type casts. Unsafe blocks are counted by traversing the High-Level Intermediate Representation (HIR) and identifying blocks marked with the unsafe keyword. For unsafe lines, we collect the set of source lines spanned by those blocks to avoid double-counting lines that fall under nested unsafe blocks. Type casts are counted by identifying explicit cast expressions in the HIR. We use a custom static analyzer built on top of the Rust compiler internals to collect these metrics. We have included this analyzer in our supplementary material [29]. As with Clippy warnings, we normalize the raw counts by the C LOC.
Results
This section presents the results and analysis from our study and also answers to our RQs. To support our analysis with code snippets, we developed a website1 to make the translated code and generated specification artifacts easily accessible.
RQ1: How does test-based validation and refinement of specifications affect specification-based translation?
a) Performance of GPT-4o-mini with Unvalidated Specifications: We first evaluate the translation performance of GPT-4o-mini with unvalidated specifications. As shown in Table I, the model demonstrates limited effectiveness, achieving a maximum of 147 successful translations in the code-and-specification setting at , and 138 and 90 at the code-only and specification-only settings, respectively. It is worth noting that while the specification-only variant produced fewer compilation errors, it did not improve correctness (pass rate); instead, it led to higher test failure rates. With this observation, we replaced GPT-4o-mini with GPT-5.2 as the model responsible for translation to target language (Rust) while retaining the same specifications generated by GPT-4o-mini. The result of this model transition is shown in Table II. This operation yielded substantial improvements, with code and specification increasing from 147 to 295, code-only successes increasing from 138 to 295, specification only increasing from 90 to 221 and at . This improvement is attributable to the stronger reasoning and code synthesis capabilities of GPT-5.2. Nevertheless, the specification-only variant still remains a limiting factor, with 75 failures at , suggesting that specification quality constrains performance independently of model capability.
| Profile | k | Total | Pass (Cum.) | Fail | RE | TO | CE |
| Code+Spec | 1 | 300 | 112 (112) | 20 | 46 | 0 | 122 |
| Code+Spec | 2 | 188 | 26 (138) | 24 | 49 | 1 | 88 |
| Code+Spec | 3 | 162 | 9 (147) | 25 | 45 | 0 | 83 |
| Code Only | 1 | 300 | 103 (103) | 8 | 41 | 2 | 146 |
| Code Only | 2 | 197 | 25 (128) | 13 | 60 | 3 | 96 |
| Code Only | 3 | 172 | 10 (138) | 19 | 42 | 4 | 97 |
| Spec Only | 1 | 300 | 45 (45) | 125 | 29 | 2 | 99 |
| Spec Only | 2 | 256 | 23 (68) | 175 | 25 | 2 | 31 |
| Spec Only | 3 | 237 | 22 (90) | 156 | 21 | 2 | 36 |
Table I. Translations with GPT-4o-mini using Specifications Generated by GPT-4o-mini
| Profile | k | Total | Pass (Cum.) | Fail | RE | TO | CE |
| Code+Spec | 1 | 300 | 279 (279) | 9 | 7 | 0 | 5 |
| Code+Spec | 2 | 21 | 16 (295) | 3 | 1 | 0 | 1 |
| Code+Spec | 3 | 5 | 0 (295) | 4 | 1 | 0 | 0 |
| Code Only | 1 | 300 | 280 (280) | 6 | 7 | 1 | 6 |
| Code Only | 2 | 20 | 11 (291) | 7 | 2 | 0 | 0 |
| Code Only | 3 | 9 | 4 (295) | 4 | 1 | 0 | 0 |
| Spec Only | 1 | 300 | 111 (111) | 160 | 26 | 0 | 3 |
| Spec Only | 2 | 189 | 74 (185) | 110 | 5 | 0 | 0 |
| Spec Only | 3 | 115 | 36 (221) | 75 | 2 | 1 | 1 |
Table II. Translations with GPT-5.2 using Specifications Generated by GPT-4o-mini
b) Performance of GPT-5.2 with Unvalidated Specifications: Based on the previous observations, we conducted an additional experiment in which GPT-5.2 is used both to generate specifications and to perform the subsequent code translation. The results of this configuration are presented in Table III. Overall, we observe trends consistent with the earlier setup. In particular, the code-and-specification and code-only variants achieve higher test pass rates (288 and 291, respectively) compared to the specification-only variant (259). The specification-only setting continues to exhibit relatively high failure rates, with 62 failures at and 34 failures at .
| Profile | k | Total | Pass (Cum.) | Fail | RE | TO | CE |
| Code+Spec | 1 | 300 | 266 (266) | 10 | 5 | 18 | 1 |
| 2 | 34 | 12 (278) | 3 | 2 | 17 | 0 | |
| 3 | 22 | 10 (288) | 2 | 3 | 7 | 0 | |
| Code Only | 1 | 300 | 275 (275) | 5 | 6 | 9 | 5 |
| 2 | 25 | 13 (288) | 6 | 2 | 3 | 1 | |
| 3 | 12 | 3 (291) | 4 | 2 | 3 | 0 | |
| Spec Only | 1 | 300 | 216 (216) | 64 | 13 | 4 | 3 |
| 2 | 84 | 34 (250) | 44 | 6 | 0 | 0 | |
| 3 | 50 | 9 (259) | 36 | 3 | 0 | 2 |
Table III. Translations with GPT-5.2 using Specifications Generated by GPT-5.2
Given the persistent pattern of high test failure rates in specification-only translation, a detailed analysis of selected cases revealed that specification incompleteness is a key limiting factor. Consider the example in Figure 2, which shows a cube root computation using Newton-Raphson iteration. The unvalidated specification (see [33]) omits the initial guess, causing the LLM to default to instead of , and describes the iteration abstractly, leading the LLM to invert the loop structure relative to the original do-while. The test-validated specification [33] explicitly corrects both omissions, yielding a correct translation. To address such cases systematically, we introduce a test-based validation and refinement step to improve specification fidelity.

Figure 2. (a) Original C source. Code comments in all panels are annotations added by the authors for readability; they were not part of the input given to the LLM, nor part of the LLM output. Amber highlights mark the two behavioral details most critical to faithful translation. (b) Differences between the unvalidated and test-validated specifications, summarized manually by the authors; the "Aspect" labels are our own categories and are not part of either specification. (c1) and (c2) show the resulting Rust translations: the unvalidated specification causes the LLM to choose an initial guess of and a different algebraic form of Newton’s method, whereas the test-validated specification yields a translation that reproduces the program’s observable behavior. Red highlights indicate deviations; green highlights indicate correct correspondence.
Based on our previous observations of LLM performance with GPT-4o-mini and GPT-5.2 on unvalidated specifications, we proceed with the rest of our experiments using GPT-5.2.
c) Performance of GPT-5.2 with Test-validated Specifications: Table IV presents the results using test-validated specifications for translation. The results indicate that specification refinement leads to substantial improvements, particularly for the specification-only variant. Specifically, when compared to the non-validated setting (Table III) for specification-only translation, the test failure rate decreases significantly from 62 to 13 at , and the number of successfully translated code instances increases from 217 to 274 (≈26% increase). After three attempts (), the failure rate further reduces from 34 to 11 for the specification-only variant. Correspondingly, the number of successfully translated programs increases from 259 to 286.
| Profile | k | Total | Pass (Cum.) | Fail | RE | TO | CE |
| Code+Spec | 1 | 300 | 281 (281) | 9 | 8 | 0 | 2 |
| 2 | 21 | 13 (294) | 7 | 1 | 0 | 0 | |
| 3 | 8 | 1 (295) | 6 | 1 | 0 | 0 | |
| Code Only | 1 | 300 | 282 (282) | 7 | 5 | 2 | 4 |
| 2 | 18 | 12 (294) | 6 | 0 | 0 | 0 | |
| 3 | 6 | 2 (296) | 4 | 0 | 0 | 0 | |
| Spec Only | 1 | 300 | 274 (274) | 15 | 7 | 0 | 4 |
| 2 | 28 | 10 (284) | 15 | 1 | 1 | 1 | |
| 3 | 18 | 2 (286) | 15 | 0 | 1 | 0 |
Table IV. Translations with GPT-5.2 using test-validated specifications generated by GPT-5.2
In the unvalidated setting, overly abstract specifications lead to plausible but incorrect translations. Refinement addresses this by adding the missing behavioral details, and some of these details are implementation-level. In the example in Figure 2, the refined specification states the initial guess and that the update precedes the stopping check, because the printed value depends on both. This is a limitation of our approach: refinement can move a specification from describing what a program guarantees toward describing how it computes it, which may in turn steer the translation back toward the original implementation. We did not measure how often this occurs. However, translations from test-validated specifications still replaced C’s algorithms or data structures in most programs (Section IV-C), suggesting that refinement did not reintroduce C’s implementation wholesale.
d) Idiomaticity of the Generated Code: Table VI presents both the raw and normalized Clippy-based idiomaticity counts across the different translation settings. Across four of the five categories, specification-only translations produce fewer warnings than both code-only and code-and-specification, suggesting that specification tends to produce more idiomatic Rust. The one exception is the correctness category, where specification-only produces 6 warnings compared to 2 for code-only and 0 for code-and-specification.
| Category | specification-only | code-only | code-and-specification |
| Correctness | (6, 0.32) | (2, 0.11) | (0, 0.00) |
| Suspicious | (0, 0.00) | (0, 0.00) | (0, 0.00) |
| Complexity | (76, 4.10) | (186, 10.04) | (114, 6.15) |
| Performance | (14, 0.76) | (30, 1.62) | (18, 0.97) |
| Style | (456, 24.61) | (1084, 58.51) | (842, 45.45) |
Table VI. Clippy warning counts (Raw, Normalized). Highlighted cells indicate the lowest normalized value per row.
A closer look at the individual warnings reveals that these correctness issues are not fundamental translation errors but artifacts of LLM-type reasoning without source code, manifesting as redundant guards that are logically vacuous in Rust’s type system. Code-only warnings, by contrast, stem from structural mimicry of C code. We note that the absolute numbers here are very small, at most 8 warnings across 900 programs, so we do not draw strong conclusions from this category alone. The overall picture from Table VI still favors specification-only and code-and-specification over code-only, particularly for the style and complexity categories which account for the large majority of warnings.
Table VII reports unsafe construct usage. Specification-only translation produces far fewer unsafe blocks (14) and lines (14) than code-only (82 blocks, 342 lines), a 6× and 24× reduction, respectively, with code-and-specification in between (22 blocks, 48 lines). The same trend holds for type casts: 691 for specification-only versus 1,756 for code-only. This is consistent with the qualitative observation in RQ3 (Section IV-C): Without direct access to C code, the model is no longer anchored to C’s low-level patterns and instead produces translations that adopt safe Rust idioms. Together, Tables VI and VII present a consistent picture that specification-based translation produces more idiomatic and safer Rust meaningfully.
To confirm that these differences are statistically meaningful, we applied the Friedman test [34] followed by post-hoc Wilcoxon signed-rank [35] tests with Holm-Bonferroni [36] correction across the 285 programs. These tests were chosen because Shapiro–Wilk [37] normality tests indicated non-normal distributions (all ), the paired nature of the data favors the Friedman test over the Kruskal–Wallis test, and the Holm-Bonferroni correction balances family-wise error control with sensitivity across three pairwise comparisons [36].
| Translation | C LOC | Rust LOC |
| specification-only | 18,527 | 24,226 |
| code-only | 18,527 | 27,842 |
| code-and-specification | 18,527 | 23,407 |
Table V. Code size of Original C and Rust code
For total Clippy warnings (, ), specification-only produced significantly fewer warnings than both code-and-specification (, , large effect) and code-only (, , large effect). The style category, which accounts for the large majority of warnings, drove this result most strongly (, ), with code-only producing significantly more style warnings than both specification-only (, large effect) and code-and-specification (, large effect). Complexity warnings followed the same pattern (, ), with code-only producing significantly more than both specification-based settings. For unsafe blocks (, ), both specification-only and code-and-specification produced significantly fewer unsafe blocks than code-only ( and , respectively, both large effects), while the difference between the two specification-based settings was not statistically significant (). Type casts showed significant differences across all three settings (, ), with code-only producing the most ( vs. specification-only, vs. code-and-specification) and specification-only the fewest.
RQ1 Answer Specification-only translation depends strongly on whether the specification passes test-based validation. Unvalidated specifications often omit behavioral details that the program’s observable behavior depends on, leading to pass rates around 86% (GPT-5.2). Our iterative back-translation refinement closes this gap, raising the specification-only pass rate to 95.3%.
RQ2: What is the impact of specifications on the trade-off between translation correctness and code quality?
Analysis of Code Quality Metrics Across Specification Sources
Table VIII presents a comparison of code quality metrics across three input configurations. To ensure that observed differences reflect genuine variation in code quality rather than differences in problem coverage, we restricted this evaluation to the intersection of problems successfully translated under all three input settings (code-only, specification-only, and code-and-specification) across the three experiment setups reported in Tables II, III, and IV. This intersection yielded a balanced subset of 193 problems on which all reported metrics were computed.
The results carry a clear message for practitioners: the context you give the model has effect on code quality attributes. Translations produced from specification-only prompts were consistently the most readable and maintainable across all three experiment setups. They used roughly 30–35% fewer logical lines of code than code-only translations (e.g., 32.90 vs. 46.90 LLOC in the first configuration) and generated 44–55% more comments (11.39 vs. 7.92 CLOC), which may be a meaningful difference for developers inheriting such code. Halstead Effort scores, which approximate the cognitive work required to understand a function, were nearly 45% lower for specification-only translations than for code-only ones (∼89,000 vs. ∼161,000–167,000). The Maintainability Index (SEI) reinforces this picture: code-only translations hovered around 38–39, falling well below the threshold of 65 associated with low maintainability [38], while specification-only translations reached 45–49, narrowing that gap considerably.
The one exception is cyclomatic complexity, where code-and-specification consistently won (13.44–14.11 vs. 15.04–16.53 for specification-only), suggesting that access to the original source helps the model preserve the branching structure of the reference implementation which is useful when control-flow conformance matters. Code-only prompting ranked the least on every other dimension, producing the largest, least commented, and most cognitively demanding output regardless of which model generated the specifications.
It is also important to note that, high structural quality does not guarantee functional correctness. Specifications generated by GPT-4o-mini produce more maintainable code but are associated with higher test failure rates when used in isolation, suggesting they omit critical behavioral details. Concise specifications may be structurally sufficient yet semantically underspecified, yielding implementations that are clean but incomplete. The validation process addresses this directly: by refining natural language descriptions, it improves both the correctness and consistency of generated code, bridging the gap between structural tidiness and functional fidelity.
Analysis of Code Quality Metrics for Test-Validated Specifications
Table IX presents code quality metrics for the test-validated specification experiment using GPT-5.2, evaluated across 285 test-passing translations drawn from Table IV. Unlike the results in Table VIII, no single modality dominates across all metrics: code-and-specification leads on structural and complexity measures, while specification-only produces more compact and better-documented output. Both, however, consistently outperform code-only, which yields the highest complexity values, the largest implementations, and the lowest maintainability score (30.94) across the board.
| Metric | Pref. | GPT-5.2 + unvalidated specs (GPT-4o-mini) | GPT-5.2 + test-validated specs (GPT-5.2) | GPT-5.2 + unvalidated specs (GPT-5.2) | ||||||
| code-and-specification | specification-only | code-only | code-and-specification | specification-only | code-only | code-and-specification | specification-only | code-only | ||
| MI (SEI) ↑ | ↑ | 44.27 | 48.57 | 38.85 | 46.03 | 44.63 | 37.86 | 45.52 | 45.39 | 38.18 |
| Cyclomatic (sum) ↓ | ↓ | 14.11 | 15.04 | 16.11 | 13.44 | 13.86 | 16.46 | 13.80 | 14.03 | 16.53 |
| Cognitive (sum) ↓ | ↓ | 18.20 | 17.19 | 20.77 | 17.11 | 16.92 | 20.45 | 17.44 | 17.12 | 21.16 |
| Halstead Volume ↓ | ↓ | 2023.19 | 1831.32 | 2411.87 | 1971.32 | 1915.54 | 2451.65 | 2017.09 | 1912.99 | 2449.90 |
| Halstead Effort ↓ | ↓ | 114766.35 | 89030.40 | 160816.71 | 108583.09 | 88795.41 | 166571.72 | 112584.04 | 89475.35 | 165057.89 |
| SLOC ↓ | ↓ | 70.25 | 69.50 | 81.70 | 67.88 | 70.01 | 82.59 | 70.29 | 70.25 | 83.86 |
| PLOC ↓ | ↓ | 53.56 | 49.37 | 63.19 | 51.33 | 51.00 | 64.12 | 52.97 | 51.14 | 65.07 |
| LLOC ↓ | ↓ | 38.59 | 32.90 | 46.90 | 37.10 | 35.48 | 47.83 | 38.08 | 34.95 | 47.45 |
| Comments (CLOC) ↑ | ↑ | 7.39 | 11.39 | 7.92 | 7.71 | 9.38 | 7.67 | 8.29 | 9.70 | 8.19 |
| Blank Lines ↓ | ↓ | 9.93 | 9.19 | 11.17 | 9.63 | 10.36 | 11.31 | 9.82 | 10.14 | 11.21 |
Table VIII. Comparison of code quality metrics across GPT-5.2-based translation configurations for test-passing translations only. ↑ indicates higher is better, and ↓ indicates lower is better. The best values per configuration are highlighted in bold. Group headers give the translation model, the specification type, and (in parentheses) the model that generated the specifications.
| Metric | Pref. | GPT-5.2 + test-validated specs (GPT-5.2) | ||
| code-and-specification | specification-only | code-only | ||
| MI (SEI) ↑ | ↑ | 38.96 | 37.89 | 30.94 |
| Cyclomatic (sum) ↓ | ↓ | 16.59 | 17.16 | 19.66 |
| Cognitive (sum) ↓ | ↓ | 21.44 | 22.45 | 24.55 |
| Halstead Volume ↓ | ↓ | 2435.78 | 2410.57 | 2960.94 |
| Halstead Effort ↓ | ↓ | 147126.90 | 131948.86 | 209261.98 |
| SLOC ↓ | ↓ | 82.13 | 85.00 | 97.69 |
| PLOC ↓ | ↓ | 62.75 | 62.81 | 76.89 |
| LLOC ↓ | ↓ | 45.84 | 44.54 | 57.29 |
| Comments (CLOC) ↑ | ↑ | 9.13 | 11.02 | 8.68 |
| Blank Lines ↓ | ↓ | 11.38 | 12.17 | 12.82 |
Table IX. Comparison of code quality metrics for Translation with Validated Specs using GPT-5.2 in test-passing translations only. ↑ indicates higher is better, and ↓ indicates lower is better. The best values are highlighted in bold.
The code-and-specification configuration achieves the highest MI (SEI) score (38.96), the lowest cyclomatic complexity (16.59) and cognitive complexity (21.44), and the smallest code size (SLOC 82.13, PLOC 62.75). This pattern is consistent with our qualitative analysis (Section IV-C): access to the original source constrains the model from introducing additional abstractions, such as defensive bounds checks, extra helper functions, and idiomatic wrappers, which can inflate both size and branching complexity. For practitioners prioritizing structural simplicity and control-flow preservation, this modality is the stronger choice.
The specification-only configuration produces the lowest Halstead Volume (2410.57), Halstead Effort (131,948.86), and logical line count (44.54), alongside the highest comment density (11.02), yielding less code, better documented, at the cost of marginally higher structural complexity. This trade-off is consistent across transformation categories, with the additional abstractions introduced by specification-only, such as defensive bounds checks and Option wrapping, explaining the slight complexity overhead. For practitioners prioritizing readability and compactness, this modality is preferable.
The choice between them, however, has modest practical consequences. The code-only baseline makes the decision easy by contrast: it produces the highest complexity values, the largest implementations, and an MI (SEI) of just 30.94, well below both specification-based configurations and far into the low-maintainability range [38], reinforcing that some form of specification is a prerequisite for structurally sound translation.
To confirm statistical significance, we applied the Friedman test [34] with post-hoc Wilcoxon signed-rank [35] tests and Holm-Bonferroni correction [36]. All metrics have significant differences, except Closures (, ).
For size metrics, code-only produced substantially more code than both specification-based settings across PLOC (), LLOC (), and SLOC () (all , large effect sizes ), while the combined and specification-only settings were largely comparable. Similarly, for complexity metrics, code-only yielded the highest Cyclomatic Complexity (), Cognitive Complexity (), and Functions (), and Function Points () (all ), with the combined approach consistently producing the lowest values. Halstead Effort () and Halstead Volume () followed the same pattern, with code-only significantly exceeding both specification-based settings (all , ). For Blank Lines (), the combined approach produced fewer than both alternatives, while specification-only and code-only did not differ significantly. The Comments metric () was the only metric in which specification-only led, producing more comments than both other settings (). For the MI (SEI) () and MI (original) (), both specification-based settings substantially outperformed code-only (large effect sizes), while the combined approach yielded modest but significantly higher maintainability than specification-only (, ).
Across all metrics, code-only consistently and substantially underperformed both specification-based approaches. The combined and specification-only settings were largely comparable, confirming that specifications are the primary driver of translation quality, with their combination with code offering only marginal additional gains in maintainability.
RQ2 Answer
Code-only translation achieves the highest correctness (98.6%) but the lowest code quality. Specification-only produces the most idiomatic and maintainable Rust, with approximately 6× fewer unsafe blocks, while code-and-specification offers the best overall balance, achieving 98.3% correctness with consistently higher-quality code.
RQ3: To what extent do specifications encourage implementation choices that are less constrained by the original source implementation?
To answer this question, we conducted a detailed manual analysis of the 285 programs that passed testing across all three settings. We examined each program by comparing the original C code with its corresponding translations, noting structural and semantic differences between the code-only and specification-based outputs. We identified recurring patterns and counted how frequently each appeared across all programs to assess whether the observations were systematic rather than incidental. This analysis reveals two consistent and complementary patterns, described in what follows.
a) What code-only translation preserves from C: When the LLM has direct access to the source code, it tends to mirror C’s implementation choices rather than producing implementations free of C’s structural choices. This manifests in several recurring ways. Variable names from the original C source, including single-letter loop counters and opaque intermediate names, are carried over into the Rust output in 98% of code-only translations, which makes the translated code harder to read. Integer types follow C conventions: i32 is used where usize or i64 would be more natural in Rust in 86% of code-only translations, and signed wrapping arithmetic is sometimes added explicitly to replicate C’s undefined behavior on overflow [39]. Loop structures, including manual index manipulation and while loops with mutable counters, are preserved rather than replaced by Rust’s iterator idioms [40] in 91% of code-only translations. Fixed-size arrays matching C’s stack-allocated dimensions appear in 70% of code-only translations in place of more flexible Vec allocations [41]. When C uses global mutable state, hand-rolled data structures, or low-level I/O patterns, the code-only translation reproduces these with varying degrees of fidelity, sometimes resorting to unsafe blocks to match C’s unguarded memory access patterns [42]. In 21.7% cases, redundant or dead variables are also reproduced in the output rather than corrected [43].
b) What specification-based translation abstracts and improves: When the LLM works from a specification rather than source code, it is no longer anchored to C’s implementation decisions and instead produces implementations that reflect the problem’s semantic rather than C’s structure. This leads to a consistent set of improvements. Algorithms are often replaced by cleaner equivalents: a hand-rolled Newton’s method square root loop becomes a single .sqrt() call [44], a manual bubble sort becomes sort_by [45], and similar substitutions occur in 90.7% of specification-only translations and 67.6% of code-and-specification translations, compared to just 1.7% of code-only translations. Data structures are chosen to match the problem’s semantics rather than C’s memory layout: parallel raw arrays become named structs with meaningful field names [46], fixed-size hash tables with manual collision handling become HashSet [47], and manual heap implementations are replaced by BinaryHeap from the standard library [42], with such upgrades appearing in 78.7% of specification-only and 66.2% of code-and-specification translations versus 4% of code-only. Type choices improve: counts described as non-negative in the specification are represented as usize or u64 rather than i32 in 98.3% of specification-only translations, and integer sentinel values like -1 become Option types [48] in 58.7% of specification-only and 45.5% of code-and-specification translations, compared to 1.3% of code-only. Unsafe code largely disappears: when the specification states that inputs satisfy certain bounds as preconditions, the LLM no longer needs to emulate C’s unguarded indexing and can use safe Rust indexing directly [39]. As shown in Table VII, specification-only translation reduces unsafe blocks by roughly 6 times compared to code-only, with code-and-specification in between. Unlike prior work such as SpecTra [17], which evaluates specifications alongside source code and measures correctness alone, these results show that specifications used in isolation can meaningfully shift translation away from C’s structural choices.
| Metric | specification-only | code-only | code-and-specification |
| Unsafe Blocks | (14, 0.76) | (82, 4.43) | (22, 1.19) |
| Unsafe Lines | (14, 0.76) | (342, 18.46) | (48, 2.59) |
| Type Casts | (691, 37.30) | (1756, 94.78) | (884, 47.71) |
Table VII. Unsafe construct counts (Raw, Normalized). Highlighted cells indicate the lowest normalized value per row.
Figure 3. Cost analysis for different translation profiles
RQ3 Answer
Specification-based translation produces implementations that are free from C’s structural choices, replacing C-style constructs with idiomatic Rust equivalents in the majority of programs: algorithm substitution occurs in 90.7% of specification-only translations versus 1.7% of code-only, and data structure upgrades appear in 78.7% versus 4%. Code-only translation, by contrast, mirrors C’s implementation choices across all categories. These results confirm that natural language specifications shift LLM translation away from syntactic mimicry toward implementations that are less constrained by the original source.
RQ4: What are the cost–quality trade-offs of specification-based translation pipelines?
Figure 3 presents the translation cost using Tiktoken [49]. The code-only profile remains stable across pipelines (4.61), while specification-only consistently incurs the highest cost due to more iterations required to produce test-passing translations.
translations. The test-validated specification pipeline substantially increases overall cost, spending $13.01 on validation alone compared to $2.85 for unvalidated specs, bringing total translation costs to 17.18 for specification-only and code-and-specification respectively.
Interpreted alongside our analysis of previous RQs, these cost differentials reveal important trade-offs. Despite its low cost ($4.45 average), code-only consistently yields the lowest maintainability and highest complexity, negating its cost advantage. The test-validated specification pipeline, despite its high cost, does not yield commensurate improvements in structural quality compared with unvalidated specifications, making GPT-5.2-generated specifications the most cost-efficient operating point. Critically, structural metrics alone do not capture translation fidelity. As discussed in Section IV-B, structurally superior translations are not necessarily functionally correct: GPT-4o-mini-based specifications produce compact, maintainable code but exhibit elevated test failure rates due to under-specification. The cost of validation should be understood as an investment in functional correctness rather than structural quality. Thus, practitioners optimizing purely for cost risk produce structurally sound but semantically deficient translations. This is important when applying specification-based translation in the context of security-critical software migration.
RQ4 Answer
The right pipeline depends on what the migration prioritizes:
Correctness and cost only: code-only (98.7% pass rate, $4.34), if code quality is not a concern; it produces the most unsafe blocks and Clippy warnings.
Correctness and code quality: code-and-specification with test-validated specifications (98.3%, $17.18), which keeps near code-only correctness with fewer unsafe blocks and warnings, at about 4× the cost.
Code quality first: specification-only with test-validated specifications (95.3%, $17.29), which produces the fewest unsafe blocks and Clippy warnings.
Limited budget: code-and-specification with unvalidated GPT-5.2 specifications (96.0%, $7.58), which gives most of the quality gains at less than half the cost of the test-validated pipeline. Specification-only should not be used without validation (86.3%).
Threats to Validity
This study has some limitations that should be considered when interpreting the results. We discuss the primary threats to internal, construct, external, and conclusion validity, together with the measures taken to mitigate their impact.
Internal Validity. Specification generation, validation, and translation are executed as independent, stateless API calls with no shared context, preventing information leakage across stages. In the specification-only setting, the model receives only the generated specification, ensuring a clean evaluation of specification sufficiency. Specification correctness is assessed through back-translation and execution against ground-truth test cases rather than LLM self-assessment. While back-translation against test cases does not guarantee full semantic completeness, this test-based validation approach is consistent with how correctness is assessed in prior work [17]. Validation and evaluation use the same test cases. Although the refinement prompt receives only a diagnosis rather than the test cases themselves, the diagnosis is derived from failing tests, and a specification is accepted only once it passes them. Test-validated specifications may therefore be tuned to the tested behaviors, which could inflate pass rates in the specification-based settings compared with a held-out test set. The Rust repair loop uses execution feedback from the same tests in all three settings, so this concern applies only to the additional specification refinement step. LLM non-determinism and the repair loop may introduce variance across runs, which we partially mitigate by evaluating two models with different capabilities.
Construct Validity. Structural quality metrics are computed using rust-code-analysis, which analyzes the Rust AST but does not capture Rust-specific semantics such as ownership, lifetimes, or trait dispatch; consequently, maintainability and complexity should be interpreted as structural approximations. Likewise, Clippy warnings serve only as a proxy for idiomaticity, which we complement with unsafe-construct metrics. Translation cost is estimated using Tiktoken rather than actual API billing, but the methodology is applied consistently across all settings, preserving the validity of relative comparisons.
External Validity. Our evaluation uses the 300-problem IBM CodeNet subset, whose programs are shorter and more self-contained than industrial C systems. Although this limits generalizability to repository-scale migration, it enables controlled comparisons and direct comparison with prior work [17]. Furthermore, our study is limited to two OpenAI models and a single language pair (C-to-Rust).
Conclusion Validity. To ensure fair comparisons, quality metrics are computed only on the balanced intersection of programs successfully translated across all three settings (193 for unvalidated specifications and 285 for test-validated specifications), thereby preventing differences in problem coverage from confounding the observed quality metrics.
CONCLUSION
This paper investigated whether natural language specifications can improve the quality of LLM-based C-to-Rust translation beyond functional correctness. We compared three translation settings (code-only, specification-only, and code-and-specification) across 300 C programs, evaluating correctness, maintainability, complexity, idiomaticity, unsafe-construct usage, and cost. Our results show a clear trade-off. Code-only translation achieves the highest correctness (98.6%) but consistently produces less maintainable, more complex, and less idiomatic Rust by mirroring C’s implementation choices. In contrast, specification-based translation frees the LLM from C’s structural choices, producing safer and more idiomatic Rust, reducing unsafe blocks by approximately and Clippy style warnings by . While specification-only translation initially lowers correctness to 86%, our iterative test-based validation and refinement process increases it to 95.3%, demonstrating that specification quality is critical to translation effectiveness. Code-and-specification provides the best overall balance, combining competitive correctness (98.3%) with substantial improvements in code quality.
Overall, our findings demonstrate that test-validated natural language specifications are an effective intermediate representation for software modernization, enabling LLMs to generate Rust that is not only functionally correct but also safer, more maintainable, and more idiomatic. Future work includes extending this approach to repository-level systems and additional source-target language pairs.
Data Availability
To enable further research and replication, the data and code used in our study are publicly available [29].
References
- [1]T. Vandervelden, R. De Smet, D. Deac, K. Steen haut, and A. Braeken, “Overview of embedded rust operating systems and frameworks,” Sensors, vol. 24, no. 17, p. 5818, 2024.DOI
- [2]P. C. van Oorschot, “Memory errors and memory safety: C as a case study,” IEEE Security & Privacy, vol. 21, no. 2, pp. 70–76, 2023.
- [3]MSRC Team, “We need a safer systems programming language,” Microsoft Security Response Center, Jul. 2019. [Online]. Available: https://www.microsoft.com/en-us/msrc/blog/2019/07/we-need-a-safer-systems-programming-language
- [4]J. Vander Stoep and C. Zhang, “Queue the hardening enhancements,” Google Online Security Blog, May 2019. [Online]. Available: https://security.googleblog.com/2019/05/queue-hardening-enhancements.html
- [5]N. D. Matsakis and F. S. Klock, “The rust language,” Ada Lett., vol. 34, no. 3, pp. 103–104, Oct. 2014. [Online]. Available: https://doi.org/10.1145/2692956.2663188
- [6]Defense Advanced Research Projects Agency, “Translating all c to rust (tractor),” https://www.darpa.mil/research/programs/translating-all-c-to-rust, 2024.
- [7]Cybersecurity and Infrastructure Security Agency and National Security Agency, “The case for memory safe roadmaps,” Cybersecurity and Infrastructure Security Agency, Tech. Rep., Dec. 2023. [Online]. Available: https://www.cisa.gov/sites/default/files/2023-12/The-Case-for-Memory-Safe-Roadmaps-508c.pdf
- [8]T. Claburn, “Microsoft is busy rewriting core windows code in memory-safe rust,” 4 2023. [Online]. Available: https://www.theregister.com/2023/04/27/microsoft_windows_rust/
- [9]“C2rust: Migrate c code to rust,” https://github.com/immunant/c2rust, 2025.
- [10]M. Emre, R. Schroeder, K. Dewey, and B. Hardekopf, “Translating c to safer rust,” Proceedings of the ACM on Programming Languages, vol. 5, no. OOPSLA, pp. 1–29, 2021.DOI
- [11]B. Tadesse, V. Nitin, M. Salah, B. Ray, M. d’Amorim, and W. Assunção, “Code quality analysis of translations from c to rust,” 2026. [Online]. Available: https://arxiv.org/abs/2602.00840
- [12]V. Nitin, R. Krishna, L. L. do Valle, and B. Ray, “C2Saferrust: Transforming C projects into safer rust with neurosymbolic techniques,” 2025. [Online]. Available: https://arxiv.org/abs/2501.14257
- [13]M. Farrukh, B. Coskun, T. Palit, and M. Polykronakis, “Safetrans: Llm-assisted translation from c to rust,” in Proceedings of the 1st Workshop on Code Translation, Transformation, and Modernization, ser. ReCode ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 30–37. [Online]. Available: https://doi.org/10.1145/3786180.3788317
- [14]H. F. Eniser, H. Zhang, C. David, M. Wang, M. Christakis, B. Paulsen, J. Dodds, and D. Kroening, “Towards translating real-world code with llms: A study of translating to rust,” 2025. [Online]. Available: https://arxiv.org/abs/2405.11514
- [15]C. Wang, T. Yu, B. Shen, J. Wang, D. Chen, W. Zhang, Y. Shi, C. Xie, and X. Gu, “Evoc2rust: A skeleton-guided framework for project-level c-to-rust translation,” 2026. [Online]. Available: https://arxiv.org/abs/2508.04295
- [16]A. Z. H. Yang, Y. Takashima, B. Paulsen, J. Dodds, and D. Kroening, “Vert: Verified equivalent rust translation with large language models as few-shot learners,” 2024. [Online]. Available: https://arxiv.org/abs/2404.18852
- [17]V. Nitin, R. Krishna, and B. Ray, “Spectra: Enhancing the code translation ability of language models by generating multi-modal specifications,” arXiv preprint arXiv:2405.18574, 2024.arxiv.org/abs/2405.18574
- [18]S. R. Dipto, S. Mondal, and C. K. Roy, “Algorithm-based pipeline for reliable and intent-preserving code translation with llms,” 2026. [Online]. Available: https://arxiv.org/abs/2602.16106
- [19]M. Shiraishi, Y. Cao, and T. Shinagawa, “Smartc2rust: Iterative, feedback-driven c-to-rust translation via large language models for safety and equivalence,” arXiv preprint arXiv:2409.10506, 2024.DOI
- [20]M. Shetty, N. Jain, A. Godbole, S. A. Seshia, and K. Sen, “Szygy: Dual code-test c to (safe) rust translation using llms and dynamic analysis,” arXiv preprint arXiv:2412.14234, 2024.arxiv.org/abs/2412.14234
- [21]C.-e. A. Tai, P. Nie, L. Golab, and A. Wong, “Nl in the middle: Code translation with llms and intermediate representations,” in 2025 IEEE International Conference on Collaborative Advances in Software and Computing (CASCON). IEEE, 2025, pp. 295–300.arxiv.org/abs/2507.08627
- [22]W. Ahmad, M. G. R. Tushar, S. Chakraborty, and K.-W. Chang, “Avatar: A parallel corpus for java-python program translation,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 2268–2281.arxiv.org/abs/2108.11590
- [23]M. Macedo, Y. Tian, F. Cogo, and B. Adams, “Exploring the impact of the output format on the evaluation of large language models for code translation,” in Proceedings of the 2024 IEEE/ACM First International conference on foundations models and software engineering, 2024, pp. 57–68.
- [24]S. Ye, Z. Sun, G. Wang, L. Guo, Q. Liang, Z. Li, and Y. Liu, “Prompt alchemy: Automatic prompt refinement for enhancing code generation,” IEEE Transactions on Software Engineering, 2025.arxiv.org/abs/2503.11085
- [25]R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker, V. Thost, L. Buratti, S. Pujar, S. Ramji, U. Finkler, S. Malaika, and F. Reiss, “Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,” 2021. [Online]. Available: https://arxiv.org/abs/2105.12655
- [26]M. Macedo, Y. Tian, P. Nie, F. R. Cogo, and B. Adams, “Intertrans: Leveraging transitive intermediate translations to enhance llm-based code translation,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 1153–1164.
- [27]Y. Ding, B. Steenhoek, K. Pei, G. Kaiser, W. L. and B. Ray, “Traced: Execution-aware pre-training for source code,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12.arxiv.org/abs/2306.07487
- [28]G. Ou, M. Liu, Y. Chen, Y. Wang, X. Peng, and Z. Zheng, “Rustre-protrans: Repository-level context code translation benchmark targeting rust,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 610–622.DOI
- [29]B. Tadesse, L. Arkoh, M. d’Amorim, and W. K. G. Assunção, “Beyond source code: Evaluating natural language specifications for llm-based c-to-rust translation,” Jun. 2026. [Online]. Available: https://doi.org/10.5281/zenodo.21080498
- [30]Mozilla, “Rust-code-analysis documentation: Supported metrics,” https://mozilla.github.io/rust-code-analysis/metrics.html.
- [31]S. Dehghan, T. Sun, W. Wu, Z. Li, and R. Jabbarvand, “Translating large-scale c repositories to idiomatic rust,” 2025. [Online]. Available: https://arxiv.org/abs/2511.20617
- [32]“Clippy linting tool,” https://github.com/rust-lang/rust-clippy, 2025.
- [33]Anonymous, “Translation example p00080,” http://74.208.64.171:8000/?variant=validated_specs_WITH_GPT_gpt-5.2&problem=p00080.
- [34]M. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,” Journal of the American Statistical Association, vol. 32, no. 200, pp. 675–701, 1937. [Online]. Available: http://www.jstor.org/stable/2279372DOI
- [35]F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945. [Online]. Available: http://www.jstor.org/stable/3001968
- [36]S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979. [Online]. Available: http://www.jstor.org/stable/4615733DOI
- [37]S. S. SHAPIRO and M. B. WILK, “An analysis of variance test for normality (complete samples),” Biometrika, vol. 52, no. 3-4, pp. 591–611, 12 1965. [Online]. Available: https://doi.org/10.1093/biomet/52.3-4.591
- [38]H. Heričko and B. Šumak, “Exploring maintainability index variants for software maintainability measurement in object-oriented systems,” Applied Sciences, vol. 13, no. 5, p. 2972, 2023.DOI
- [39]Anonymous, “Translation example p00013,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00013.
- [40]———, “Translation example p00015,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00015.
- [41]———, “Translation example p00071,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00071.
- [42]———, “Translation example p00244,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00244.
- [43]———, “Translation example p00023,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00023.
- [44]———, “Translation example p00010,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00010.
- [45]———, “Translation example p00124,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00124.
- [46]———, “Translation example p00453,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00453.
- [47]———, “Translation example p00949,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00949.
- [48]———, “Translation example p00261,” http://74.208.64.171:8000/translations?set=translation_with_validated_specs_GPT_5_2&problem=p00261.
- [49]OpenAI, “tiktoken,” https://github.com/openai/tiktoken.

