WorksComputer Science
SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code
Abstract
Understanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes.
Introduction
Understanding how code changes affect program semantics is a core aspect of code reasoning in large language models (LLMs). Evaluating this understanding has received increasing attention (Wei et al., 2025). One common test asks a model whether two programs are semantically equivalent, that is, whether they produce the same output for every input [27]. Existing benchmarks commonly construct such program pairs from seed programs by applying small automated code changes, called transformations, that either preserve or alter a program’s semantics [19, 27] (Wei et al. 2025). In the simplest case, only a single transformation is applied [19], so the evaluation does not test a model as transformations accumulate. CETBENCH addresses this by composing several transformations into one transformation sequence [27], but it still asks only whether the two programs are semantically equivalent. A yes/no answer, however, does not reveal how the model arrived at it. A model may correctly judge two programs as non-equivalent without knowing which transformation is responsible for the semantic difference or which sequence of transformations produced the modified program from the original. Equivalence accuracy alone therefore cannot measure this finer-grained reasoning. Measuring it requires asking the model these questions directly.
In this paper, we go a step further: beyond asking whether two programs are semantically equivalent, we ask which transformation changed their semantics and in which order the transformations were applied. To this end, we define three tasks on pairs of an original program and a modified program produced by a sequence of transformations. (T1) Detecting semantic equivalence: decide whether the two programs are semantically equivalent. (T2) Diagnosing semantic change: given a non-equivalent pair, identify which transformation altered the semantics. (T3) Reconstructing transformation sequences: recover the ordered sequence of transformations that produces the modified program from the original one. To investigate LLM performance on these tasks, we build SCRBench from 418 MBPP programs ([2]). We use each MBPP program as an original program and apply to it a sequence of one to five transformations from a catalog of nine semantics-preserving and semantics-altering transformations (Table 1). In every non-equivalent pair, exactly one of the applied transformations alters semantics, so T2 has a single correct answer. Every pair carries ground truth for all three tasks (§ 2). SCRBench contains 67,658 program pairs. From it we draw a stratified subset of 4,500 pairs to evaluate all three tasks as transformations accumulate: for each number of applied transformations from one to five, it contains 450 equivalent and 450 non-equivalent pairs. T1 and T3 use all 4,500 pairs, and T2, which applies only to non-equivalent pairs, uses the 2,250 non-equivalent ones. We evaluate five language models on the stratified subset with a prompt that includes chain-of-thought reasoning and a one-shot example. Within each task, we analyze how accuracy changes as transformations accumulate (§ 3.2–§ 3.4). We also perform an ablation study on a single model to assess the impact of chain-of-thought reasoning and the one-shot example on accuracy (§ 3.2–§ 3.4). Finally, we analyze the results across the three tasks to characterize the progression of code reasoning, including the opportunities it reveals for improving advanced models (§ 3.5). Detecting semantic equivalence (T1) and diagnosing semantic change (T2) show substantial variation across models, and T2 reveals differences that T1 hides. For example, Mistral-Small-24B is within five percentage points of the strongest models on T1 (94.71%) but about 40 points behind them on T2 (54.44%). Across all models, T1 accuracy ranges from 99.69% for Qwen3-235B to 55.49% for GPT-4.1-nano, and the strongest models remain accurate as transformations accumulate, while weaker models decline. T2 accuracy ranges from 97.07% for GPT-OSS-120B to 22.67% for Llama-3.1-8B. Reconstructing transformation sequences (T3) is more demanding, because it requires recovering the ordered sequence of transformations. Even models that are accurate on T1 and T2 show substantially lower accuracy on T3. GPT-OSS-120B, the strongest model on T3, reconstructs 39.27% of the sequences, and no model exceeds 1.11% when five transformations are applied. Qwen3-235B, the most accurate model on T1 (99.69%), reconstructs only 26.82% of the sequences.
| ID | Abbr. | Type | Effect |
| 1 | AOM | Altering | Swaps and . |
| 2 | BLM | Altering | Negates the first True (else the first False). |
| 3 | LOM | Altering | Swaps andor. |
| 4 | COM | Altering | Flips each comparison’s leading operator (, , ). |
| 5 | RV | Preserving | Renames the most frequent variable to VAR_0. |
| 6 | DCI_V1 | Preserving | Inserts an unused assignment. |
| 7 | DCI_V2 | Preserving | Inserts a never-executed if block. |
| 8 | OS | Preserving | Swaps the operands of a comparison between two variables (). |
| 9 | LT | Preserving | Rewrites a for-range loop as a while loop. |
Table 1. The nine transformations in the catalog .
Our contributions are twofold: (i) We study two tasks beyond detecting semantic equivalence (T1), diagnosing semantic change (T2) and reconstructing transformation sequences (T3), and show with five language models that T1 accuracy can overstate how well a model understands the effect of code changes on program semantics: T1 exposes weaker models, T2 separates models that perform similarly on T1, and models near the ceiling on T1 and T2 still struggle on T3 (§ 3). (ii) To support this study, we build SCRBench, 67,658 program pairs with automatically checked ground truth for all three tasks: an equivalence label, the single semantics-altering transformation of each non-equivalent pair, and every transformation sequence connecting the two programs, so that any valid order counts as correct (§ 2); we will publicly release SCRBench and our code upon conference acceptance.
Data Generation
We build SCRBench, a dataset of program pairs derived from MBPP programs. Each pair is represented as , where is an original program from MBPP and is the modified program obtained by applying a sequence of one to five deterministic transformations to . Each pair is annotated with a label indicating whether the two programs are semantically equivalent and with the set of all valid transformation sequences that produce from . § 2.1 covers the preprocessing of MBPP into a base corpus of original programs. § 2.2 introduces the transformations and defines valid transformation sequences and, for each pair, the transformation sequence set and length (Definitions 1–2). § 2.3 describes how the transformations are applied to generate the modified programs. Section 2.4 describes how the generated sequences form labeled pairs and reports the resulting corpus and the stratified subset drawn from it.
Base corpus
MBPP (Austin et al., [2]) is a crowd-sourced dataset of short, introductory Python programs collected for program synthesis, each paired with assert-based unit tests. We pool the train, validation, and test splits of the sanitized, hand-verified subset of MBPP (420 programs) and preprocess them in three steps. First, we canonicalize each program by parsing it to an abstract syntax tree (AST) and unparsing it. The transformations also produce code in this canonical form, so an original program and its modified program differ only in the applied transformations, not in formatting or comments. Second, we add one hand-written edge-case assertion to each of eight programs whose test suites lacked one, so that the tests also catch semantic changes that only affect edge-case inputs. Third, we remove the two programs whose canonical form duplicates another program. This yields a base corpus of 418 programs. Each entry comprises an original program and its test suite .
Transformations
A transformation is a deterministic function that rewrites a program. It is semantics-altering if it is designed to change the program’s semantics and semantics-preserving otherwise. Table 1 lists the nine transformations we use, adapted from Maveli et al. [19]: four altering (, ids 1–4) and five preserving (, ids 5–9), which together form the catalog . Appendix § 6.§ 1 motivates this choice, and Appendix § 6.§ 2 gives an example of each.
To let transformations accumulate, we apply them in sequence. Applying a transformation sequence to an original program produces the programs and ends in the modified program . We keep only sequences that meet three conditions, formalized in Definition 1. First, each transformation appears at most once, because applying a transformation twice can undo it. For example, a second AOM swaps every back to . Second, at most one transformation is altering, so that T2 has a single correct answer. Third, every transformation has its intended effect, and the tests confirm it. A transformation can leave the program unchanged, for example AOM on a program without arithmetic operators. An altering transformation can also change only code that the tests never execute, so the tests cannot confirm the semantic change. A transformation can also make a program run forever, for example when AOM turns a loop increment into a decrement, so we also reject a transformation if the tests on its output exceed a time limit.
Definition 1 (Valid transformation sequence). A transformation sequence applied to with tests is valid iff (1) its transformation ids are pairwise distinct, (2) at most one is altering, and (3) every is valid: it changes the AST of , the tests on finish within a time limit, and passes all of them if are all preserving and fails at least one otherwise.
Different valid sequences can produce the same pair, for example the same transformations applied in a different order. We therefore annotate each pair with all of them. Appendix § 6.§ 3 shows a pair with two such sequences. The length of these sequences is the number of transformations that separate from . We use it to stratify the subset and to report how accuracy changes as transformations accumulate.
Algorithm 1 BUILDCOLLECTIONS — construct every from its parent collection.
1: Input: base corpus 2: Output: collections for , 3: for to do 4: for to do 5: if then 6: ; 7: else if then 8: ; 9: else 10: ; 11: 12: return
Definition 2 (Sequence set and length). For a pair , the transformation sequence set is the set of all valid transformation sequences of length at most five that produce from . All sequences in have the same length, which Appendix § 6.§ 5 verifies for every pair in SCRBench; this length is the transformation sequence length .
Incremental generation
We now generate program pairs from the base corpus. The first transformations of a sequence form a prefix, which produces . We group prefixes into collections by their length and by the position of their altering transformation, with if they contain none. The base corpus is . Algorithm 1 builds each collection by extending the prefixes of a parent collection with transformations from a pool, either or .
The algorithm distinguishes three cases. If , it appends a preserving transformation to a prefix of , which contains no altering transformation. If , it appends an altering transformation to a prefix of . If , it appends a preserving transformation to a prefix of , which already contains its altering transformation. Every prefix therefore contains at most one altering transformation. EXTEND appends each transformation of the pool that is not yet in the prefix and marks the new prefix as valid or invalid by Definition 1. It extends only valid prefixes, because a sequence with an invalid prefix is itself invalid, so no valid sequence is lost. Appendix § 6.§ 4 gives the full procedure.
Corpus
Each valid prefix of length is a valid transformation sequence from to a program . We group these sequences by the pair that they produce and record each pair with its transformation sequence set . The pair is labeled non-equivalent () if a sequence in contains an altering transformation, and equivalent () otherwise. Appendix § 6.§ 5 also verifies that either all sequences in contain an altering transformation or none does, so is well-defined.
From the 418 original programs, we obtain 67,658 labeled pairs, of which 12,510 are equivalent and 55,148 are non-equivalent. Table 6 in Appendix § 6.6 breaks them down by transformation sequence length . The pairs are unevenly distributed: most are non-equivalent, and 89% have . To weigh every length and label equally, we use stratified sampling: we split the pairs into ten strata, one for each combination of and label, and draw the same number of pairs from each stratum. The smallest stratum, the non-equivalent pairs at , limits this number, and we draw 450 pairs per stratum. The resulting stratified subset contains 4,500 pairs, 2,250 equivalent and 2,250 non-equivalent. Within each stratum, we choose the pairs so that even the least frequent transformation appears in as many pairs as possible (Appendix § 6.7).
| Equivalent pairs | Non-equivalent pairs | Total | |
| 1 | 1,315 | 464 | 1,779 |
| 2 | 2,958 | 2,818 | 5,776 |
| 3 | 4,098 | 9,889 | 13,987 |
| 4 | 2,760 | 20,367 | 23,127 |
| 5 | 1,379 | 21,610 | 22,989 |
| Total | 12,510 | 55,148 | 67,658 |
Table 6. Number of labeled pairs by transformation sequence length .
Experiments
We evaluate five language models on the three tasks defined in § 1, using labeled program pairs from SCRBench. § 3.1 describes the common experimental setup. § 3.2–§ 3.4 report the results for T1, T2, and T3, respectively, and § 3.5 examines the progression of code reasoning capabilities across tasks.
Experiment Setup
For each task, we first compare four prompt variants on a single model, Qwen3-235B, and then compare five models using a prompt with chain-of-thought reasoning and a one-shot example. In both, we report overall accuracy and accuracy by transformation sequence length (§ 2).
Models. We evaluate five language models: Qwen3-235B (Yang et al., 2025), an open-weight mixture-of-experts model with 22B active parameters; GPT-OSS-120B (OpenAI, [25]), an open-weight mixture-of-experts reasoning model; Mistral-Small-24B (Mistral AI, [20]) and Llama-3.1-8B (Grattafiori et al., [8]), two open-weight dense models; and GPT-4.1-nano (OpenAI, [26]), a proprietary model of undisclosed size. Appendix C lists the exact model identifiers.
Prompting. We construct four prompt variants by independently adding chain-of-thought (CoT) reasoning and a one-shot example to the base instruction: Base, Base+CoT, Base+one-shot, and Base+CoT+one-shot (Table 7; templates in Appendices B.2–B.4).
| Variant | Answer instruction | Example |
| Base | direct | format-only† |
| Base+CoT | CoT | format-only |
| Base+one-shot | direct | one-shot |
| Base+CoT+one-shot | CoT | one-shot |
Table 7. Composition of the four prompt variants.
Model execution and response processing. We perform inference at temperature 0 with a maximum generation budget of 4,000 tokens and obtain one response for each program pair. A rule-based parser extracts the model’s answer from each response. Responses that match no parsing rule are classified as unparseable and counted as incorrect.
T1: Detecting Semantic Equivalence
In T1, the model receives an original program and a modified program and predicts whether they are semantically equivalent. The ground-truth label is from § 2.4. The parser extracts either YES or NO from the response. We evaluate T1 on all 4,500 pairs of the stratified subset.
Table 2 reports T1 accuracy. For Qwen3, prompt choice substantially affects T1 accuracy: Base+CoT+one-shot performs best and Base+one-shot worst, with a 13.96-percentage-point gap. Adding CoT raises T1 accuracy by 11.58 percentage points without the one-shot example (from 88.02% to 99.60%) and by 13.96 percentage points with it (from 85.73% to 99.69%). Once CoT is included, the one-shot example has almost no effect, improving accuracy by only 0.09 percentage points; without CoT, it lowers accuracy by 2.29 percentage points. These results indicate that adding the CoT instruction substantially improves T1 performance.
| Model | Prompt | Overall | |||||
| Qwen3-235B | Base | 88.02% | 98.33% | 91.78% | 88.11% | 82.56% | 79.33% |
| Qwen3-235B | Base+CoT | 99.60% | 100.00% | 99.56% | 99.56% | 99.56% | 99.33% |
| Qwen3-235B | Base+one-shot | 85.73% | 97.44% | 89.67% | 87.56% | 78.89% | 75.11% |
| Qwen3-235B | Base+CoT+one-shot | 99.69% | 99.89% | 99.22% | 99.56% | 99.89% | 99.89% |
| GPT-OSS-120B | Base+CoT+one-shot | 98.16% | 99.22% | 98.56% | 98.67% | 97.22% | 97.11% |
| Mistral-Small-24B | Base+CoT+one-shot | 94.71% | 98.89% | 95.56% | 93.56% | 93.67% | 91.89% |
| Llama-3.1-8B | Base+CoT+one-shot | 67.93% | 78.56% | 70.78% | 67.44% | 62.22% | 60.67% |
| GPT-4.1-nano | Base+CoT+one-shot | 55.49% | 58.00% | 55.56% | 54.00% | 56.22% | 53.67% |
Table 2. T1 accuracy across prompt and model variants.
Under the designated Base+CoT+one-shot prompt, accuracy is generally highest at and tends to decrease as increases. Qwen3 remains near ceiling, varying by at most 0.67 percentage points across sequence lengths, whereas Llama-3.1 declines by 17.89 percentage points, from 78.56% to 60.67%. This pattern indicates that longer transformation sequences are associated with lower semantic-equivalence accuracy, particularly for weaker models. This pattern is consistent with Laneve et al. [15], who find that combining two semantics-preserving transformations lowers equivalence accuracy below that of either transformation alone. T1 therefore measures not only whether models can detect semantic equivalence, but also how robustly they maintain that judgment as transformation sequences become longer. Performance on single-transformation pairs can overstate a model’s ability to reason about equivalence under more complex transformation sequences.
For some weaker models, overall accuracy can also obscure class-specific bias. For example, GPT-4.1-nano correctly classifies only 11.64% of equivalent pairs but 99.33% of non-equivalent pairs, indicating that its aggregate performance is largely driven by predicting NO. Because the one-shot example also has a NO answer, we additionally evaluate this model with the CoT prompt without the one-shot example to examine whether the example contributes to this bias. Removing the example raises overall accuracy from 55.49% to 64.69%, but the class-specific accuracies remain imbalanced at 31.16% and 98.22% for equivalent and non-equivalent pairs, respectively.
Taken together, these results expose two limitations of weaker models on T1: accuracy can decline as transformation sequences grow, and aggregate accuracy can be driven by class-biased predictions. For stronger models, the limitation lies in T1 itself: near-ceiling accuracy does not by itself establish that they can reason about the semantic effects of individual transformations, because T1 requires only a binary equivalence judgment; T2 addresses this by asking which transformation altered the semantics.
T2: Diagnosing Semantic Change
Given a non-equivalent program pair and the transformation catalog (see Appendix § 7.§ 1), T2 requires the model to identify which transformation is responsible for the semantic difference between the programs. We evaluate T2 on the 2,250 non-equivalent pairs of the stratified subset.
Table 3 reports T2 accuracy. CoT substantially improves T2 accuracy: on Qwen3, it raises accuracy from 43.51% to 93.51% without the one-shot example and from 37.29% to 93.47% with it. The one-shot example does not improve T2 accuracy: it lowers accuracy from 43.51% to 37.29% without CoT and from 93.51% to 93.47% with CoT.
| Model | Prompt | Overall | |||||
| Qwen3-235B | Base | 43.51% | 93.78% | 52.44% | 29.11% | 24.00% | 18.22% |
| Qwen3-235B | Base+CoT | 93.51% | 100.00% | 92.67% | 92.22% | 91.33% | 91.33% |
| Qwen3-235B | Base+one-shot | 37.29% | 96.89% | 45.56% | 20.44% | 15.11% | 8.44% |
| Qwen3-235B | Base+CoT+one-shot | 93.47% | 100.00% | 93.11% | 90.67% | 91.56% | 92.00% |
| GPT-OSS-120B | Base+CoT+one-shot | 97.07% | 100.00% | 95.33% | 96.22% | 95.56% | 98.22% |
| Mistral-Small-24B | Base+CoT+one-shot | 54.44% | 94.00% | 58.00% | 44.00% | 41.33% | 34.89% |
| Llama-3.1-8B | Base+CoT+one-shot | 22.67% | 70.89% | 24.67% | 10.67% | 3.78% | 3.33% |
| GPT-4.1-nano | Base+CoT+one-shot | 61.07% | 87.78% | 67.56% | 56.22% | 49.56% | 44.22% |
Table 3. T2 accuracy across prompt and model variants.
All models perform best at . T2 accuracy declines modestly for GPT-OSS and Qwen3 but sharply for the other three models, most of all for Llama-3.1 (from 70.89% to 3.33%), indicating that diagnosis becomes more difficult as semantics-preserving transformations accumulate around it.
Table 4 shows that T2 performance varies across models and types of semantic alteration. Higher-performing models remain accurate across alteration types, whereas lower-performing models show substantial variation and make different transformation-specific confusions. For example, Llama-3.1 often predicts preserving transformations such as RV or DCI_V2, whereas GPT-4.1-nano frequently confuses operator-altering transformations such as AOM and COM.
| Model | AOM | BLM | LOM | COM |
| Qwen3-235B | 93.98% | 95.93% | 99.77% | 86.85% |
| GPT-OSS-120B | 93.57% | 99.10% | 100.00% | 97.65% |
| Mistral-Small-24B | 56.77% | 22.17% | 62.10% | 68.86% |
| Llama-3.1-8B | 28.73% | 6.79% | 6.85% | 37.56% |
| GPT-4.1-nano | 65.25% | 43.67% | 51.83% | 74.65% |
Table 4. T2 accuracy by altering transformation using the designated prompt Base+CoT+one-shot.
Because T1 records only a binary equivalence label, it does not reveal which transformation a model attributes the change to; T2 evaluates that attribution directly. Thus, T2 reveals limitations that equivalence checking alone cannot capture. However, T2 concerns only the single transformation
that altered the semantics; we address this limitation in T3, which evaluates the full transformation sequence.
T3: Reconstructing Transformation Sequences
In T3, given , , and the transformation catalog , the model predicts an ordered sequence of transformations that maps to . A prediction is correct if it is one of the sequences in . Because is not disclosed in the prompt, the model must infer the sequence length as well as identify and order the transformations. Reconstructing this transformation history provides a controlled test of change tracking, an ability relevant to software-engineering tasks such as debugging, program repair, and code review. We evaluate T3 on all 4,500 pairs of the stratified subset.
Table 5 reports T3 accuracy. Prompt variants have limited effect on overall T3 accuracy, indicating that these prompt changes alone do not resolve the difficulty of sequence reconstruction. Accuracy decreases sharply as increases for every model. Because T3 requires the entire sequence to be correct, an error in any transformation, its order, or the inferred length makes the prediction incorrect; as sequences become longer, there are more opportunities for such errors.
| Model | Prompt | Overall | |||||
| Qwen3-235B | Base | 25.42% | 78.89% | 38.22% | 7.78% | 2.11% | 0.11% |
| Qwen3-235B | Base+CoT | 27.93% | 99.33% | 32.11% | 7.78% | 0.33% | 0.11% |
| Qwen3-235B | Base+one-shot | 22.47% | 75.44% | 27.22% | 8.56% | 1.11% | 0.00% |
| Qwen3-235B | Base+CoT+one-shot | 26.82% | 99.00% | 28.56% | 5.89% | 0.67% | 0.00% |
| GPT-OSS-120B | Base+CoT+one-shot | 39.27% | 99.22% | 62.56% | 26.22% | 7.22% | 1.11% |
| Mistral-Small-24B | Base+CoT+one-shot | 25.11% | 85.11% | 35.22% | 4.44% | 0.78% | 0.00% |
| Llama-3.1-8B | Base+CoT+one-shot | 3.09% | 14.22% | 1.00% | 0.22% | 0.00% | 0.00% |
| GPT-4.1-nano | Base+CoT+one-shot | 19.96% | 71.67% | 23.44% | 4.00% | 0.67% | 0.00% |
Table 5. T3 accuracy across prompt and model variants.
The contrast with T1 and T2 is substantial: models that perform strongly at detecting semantic equivalence and diagnosing the semantics-altering transformation still struggle to reconstruct how the modified program was produced. Thus, success on the preceding tasks does not imply that a model can recover the complete transformation sequence.
A Progression of Code Reasoning Capabilities
The three tasks form a progression from detecting semantic equivalence (T1), the standard task, to the two tasks we introduce: diagnosing semantic change (T2) and reconstructing transformation sequences (T3). Figure 1 presents the models’ overall accuracy on each task using Base+CoT+one-shot; each task reveals limitations that the tasks before it do not.
Figure 1. Overall accuracy on all tasks for all models, using Base+CoT+one-shot.
T2 distinguishes models that T1 does not. On T1, Mistral-Small-24B is within five percentage points of the strongest models (94.71%), but on T2 it reaches only 54.44%, while Qwen3-235B and GPT-OSS-120B remain above 93%. The ranking also changes: Qwen3 is the most accurate model on T1, whereas GPT-OSS is the most accurate on T2. High equivalence accuracy therefore does not imply that a model can identify which change altered the semantics.
T3 remains difficult even for models that succeed on both T1 and T2. Qwen3 and GPT-OSS exceed 93% on T1 and T2, yet reconstruct only 26.82% and 39.27% of sequences, and every model is substantially less accurate on T3 than on either preceding task. Identifying the altering transformation is thus much easier than recovering the complete ordered sequence.
Together, T2 and T3 complement equivalence checking: T2 exposes differences in attributing semantic change that T1 accuracy hides, and T3 marks a frontier that even the strongest models have not reached, namely tracking every transformation
and its order.
Related Work
There are three lines of research adjacent to our work: (i) benchmarks that pose the semantic equivalence detection question on transformed program pairs; (ii) studies of LLM robustness under semantics-preserving perturbations of code; and (iii) classical, formal techniques for program equivalence checking.
Benchmarking LLM-based code equivalence detection. The prior work closest to SCRB Bench builds pair datasets by applying transformations to seed programs and asking a language model whether the two programs are equivalent. Seq-CoBench [19] applies a single semantics-preserving or altering transformation to MBPP programs [2]. CETBench [27] applies a random sequence of up to five perturbations to pairs of CodeContests solutions and reports a substantial drop in LLM accuracy on the perturbed pairs. EquiBench (Wei et al., 2025) covers Python, C, CUDA and x86-64 and generates pairs via program analysis, compiler scheduling, and superoptimization. Allamanis and Yin [1] use LLMs with execution feedback to disprove equivalence via counterexample search. Our work differs from these studies on two axes. First, transformations in SCRBench are composed sequentially on a single seed, whereas SEQCOBENCH applies a single transformation and CETBENCH distributes each perturbation over one of two programs in a starting pair. Second, in addition to detecting semantic equivalence (T1), we evaluate LLMs on diagnosing which transformation in the sequence changed the semantics (T2) and on reconstructing the sequence that produces the target from the source (T3). To our knowledge no prior benchmark in this family addresses either.
One of the major applications of semantic equivalence detection techniques is equivalent mutant identification. Mutation testing (Jia and Harman [13]; Papadakis et al. [29]) inserts small syntactic faults into a program to produce mutants and measures how many of the mutants are killed by the test suite. An obstacle to mutation testing is the equivalent mutant problem: a syntactically distinct mutant may nonetheless be semantically equivalent to the original and therefore unkillable, biasing the mutation score (Madeyski et al. [18]). Classical equivalent mutant identification techniques include compiler-optimization checks (Offutt and Craft [24]; Papadakis et al. [28]; Kintis et al. [14]), program slicing (Hierons et al. [11]), and constraint and symbolic execution based reasoning (Nica and Wotawa [23]); learning-based approaches include feature-engineered classifiers (Naeem et al. [21]), and AST-based neural models (Peacock et al. [30]). More recently LLMs are applied directly to solve this problem (Tian et al. [33]), typically on benchmarks like MUTANTBENCH (van Hijfte and Oprescu (2021)).
Semantics-preserving perturbations and model robustness. A parallel line studies LLM robustness on downstream code tasks under semantics-preserving perturbations of the input. RECODE (Wang et al. (2023)) perturbs docstrings, function names, syntax, and format on HumanEval (Chen et al. [5]) and MBPP, whose test suites EvalPlus later strengthened (Liu et al. [17]). Henkel et al. [10] use sequences of semantics-preserving transformations to attack and harden code-summarization models, and Hooda et al. [12] apply counterfactual mutations such as branch flipping to probe conceptual understanding.
Laneve et al. [15] apply compiler-style transformations such as copy propagation and constant folding and find that models often fail to recognize the rewritten program as still equivalent. Related studies explore LLM code reasoning by asking the model to predict inputs or outputs of a given program (Gu et al., [9]; Liu et al., [16]). All of these works measure how a fixed downstream prediction shifts when the input is rewritten, whereas SCRBench treats the transformations themselves as first-class objects the model must reason about: whether a chain of transformations alters semantics, which one does, and in what order the transformations are applied.
Formal program equivalence detection. Program equivalence is undecidable in general, yet a rich body of research develops sound techniques for restricted settings: translation validation for optimizing compilers (Pnueli et al. [32]; Necula [22]), differential symbolic execution (Person et al. [31]), and semantic program alignment (Churchill et al. [6]). Badihi et al. [3] released EQBENCH, a benchmark for evaluating such tools. These formal techniques are typically sound but incomplete and scale poorly with syntactic diversity. The recent LLM-based techniques provide the opportunity for large scale program equivalence analysis, which we study through T1–T3, using tens of thousands of labeled pairs in SCRBench with various levels of syntactic differences.
Conclusion
Semantic equivalence checking is a common test of how language models reason about code changes, but a yes/no answer does not reveal whether a model understands the semantic effect of each change. We therefore studied two tasks beyond (T1) detecting semantic equivalence: (T2) diagnosing semantic change, that is, identifying which transformation altered the semantics, and (T3) reconstructing transformation sequences, that is, recovering the ordered transformations that produced the modified program. To evaluate language models on these three tasks, we built SCRBench, a dataset of 67,658 Python program pairs generated by applying sequences of semantics-preserving and semantics-altering transformations to 418 MBPP programs, with ground truth for all three tasks; SCRBench can also support training models on them. For weaker models, T1 is informative, particularly as transformations accumulate and their accuracy declines. For stronger models, however, T1 accuracy approaches the ceiling, giving the misleading impression that these models fully understand how code changes affect program semantics. The two new tasks expose what T1 hides: T2 separates models that perform similarly on T1, and T3 remains difficult even for the strongest models, the best of which reconstructs only 39.27% of sequences. Semantic equivalence checking is therefore a useful but incomplete measure of code reasoning. T2 and T3 show where even strong models can still improve: attributing and tracking code changes, as needed in debugging, program repair, and code review.
Limitations
SCRBench consists of short Python functions from MBPP and a catalog of nine transformations adapted from SEQCOBENCH, so it does not cover larger programs, other languages, or more complex transformations. The equivalence labels also depend on tests: a pair is labeled equivalent if every applied transformation is designed to preserve semantics and the MBPP tests still pass, which does not prove equivalence. The non-equivalent labels are certain, because a failing test shows that the two programs behave differently. The cross-model comparison uses a single prompt variant with one response per pair at temperature 0. Extending the program sources and the transformation catalog, and evaluating additional models and prompting strategies, are directions for future work.
Ethics Statement
SCRBench is derived from MBPP (Austin et al., [2]), which is released under the CC BY 4.0 license (Google Research, [7]), and contains only short, self-contained Python programming problems written for benchmarking. It contains no personal data, and our study involves no human subjects. All models are accessed through public APIs under their providers’ terms of use. We do not foresee direct negative societal impact; the benchmark is intended to measure, and ultimately improve, how reliably language models reason about the semantic effect of code changes.
Data generation details
This appendix gives the details of the data generation in § 2, from the choice of transformations to the stratified subset.
Choice of transformations
We adapt the catalog from SEQCOBENCH ([19]), which keeps our results comparable with prior code-equivalence work. We omit its Identity-Operator-Misuse category, which changes only 3 of the 418 base programs, and its Dissimilar-Code-Selection category, which does not test reasoning about individual transformations.
The preserving transformations follow common practice. NatGen uses variable renaming, dead-code insertion, operand swapping, and for-to-while conversion ([4]), and RV and DCI_V2 follow its VAR_i naming and always-false guards. AOM, LOM, and COM are restricted forms of the AOR, LCR, and ROR mutation operators ([29]), a standard way to inject bugs in software testing; they cover common operator mistakes, not all kinds of real-world bugs.
Single-transformation examples
This subsection illustrates each transformation in Table 1 with a program from the base corpus. Each example applies the transformation once to an original program.
AOM, LOM, COM, and OS change every matching location in the program and LT rewrites every range loop not nested in another for loop, whereas BLM, RV, DCI_V1, and DCI_V2 make a single edit. Within a transformation sequence, each transformation applies to the current program, including code inserted by earlier transformations.
AOM (id 1, altering). Swaps + with - and * with /. In the example, all three operators change, so the function no longer computes hexagonal numbers.
Before
def hexagonal_num(n):
return n * (2 * n - 1)
After
def hexagonal_num(n):
return n / (2 / n + 1)
BLM (id 2, altering). Flips the first True to False or, if the program contains no True, the first False to True. In the example, the function now returns False for every input.
Before
def dif_Square(n):
if n % 4 != 2:
return True
return False
After
def dif_Square(n):
if n % 4 != 2:
return False
return False
LOM (id 3, altering). Swaps and with or. In the example, the base case can no longer be reached, so every call recurses until Python raises a RecursionError.
Before
def newman_prime(n):
if n == 0 or n == 1:
return 0
return 2 * newman_prime(n - 1) + newman_prime(n - 2)
After
def newman_prime(n):
if n == 0 and n == 1:
return 1
return 2 * newman_prime(n - 1) + newman_prime(n - 2)
COM (id 4, altering). Swaps the leading operator of each comparison: < and >, <= and >=, and == and !=. In the example, the sign test is reversed.
Before
def opposite_Signs(x, y):
return x ^ y < 0
After
def opposite_Signs(x, y):
return x ^ y > 0
RV (id 5, preserving). Renames the most frequently used variable or parameter to VAR_0. In the example, XOR is renamed.
Before
def search(arr):
n = len(arr)
XOR = 0
for i in range(n):
XOR = XOR ^ arr[i]
return XOR
After
def search(arr):
n = len(arr)
VAR_0 = 0
for i in range(n):
VAR_0 = VAR_0 ^ arr[i]
return VAR_0
DCI_V1 (id 6, preserving). Inserts, at the start of the most deeply nested block, an assignment of a constant expression to a new variable that is never read. The variable names come from a fixed vocabulary; during preprocessing we verified that none of them occurs in any original program. In the example, the function body is the most deeply nested block, so the assignment to window_scope is inserted before the return.
Before
def count(lst):
return sum(lst)
After
def count(lst):
window_scope = 2 ** 20 >> 10 ^ 42, 'ab'[0] + chr(48),
↪ (3, 5, 7)[1]
return sum(lst)
DCI_V2 (id 7, preserving). Inserts, at the same location as DCI_V1, an if block whose condition is always false (the left-hand side evaluates to ). The block contains the same kind of unused assignment and an empty try/except. In the example, the block is inserted before the return, where DCI_V1 inserts its assignment.
Before
def count(lst):
return sum(lst)
After
def count(lst):
if sum((13, 7, 11)) ^ 27 == 0:
window_scope = 2 ** 20 >> 10 ^ 42, 'ab'[0] +
↪ chr(48), (3, 5, 7)[1]
try:
pass
except:
pass
return sum(lst)
OS (id 8, preserving). Swaps the operands of each <, <=, >, or >= comparison between two variables and mirrors the operator. In the example, a >= b becomes b <= a.
Before
def maximum(a, b):
if a >= b:
return a
else:
return b
After
def maximum(a, b):
if b <= a:
return a
else:
return b
LT (id 9, preserving). Rewrites a for loop over range as an equivalent while loop with an explicit counter. In the example, the counter i is initialized to 0 before the loop and incremented at the end of its body.
Before
def search(arr):
n = len(arr)
XOR = 0
for i in range(n):
XOR = XOR ^ arr[i]
return XOR
After
def search(arr):
n = len(arr)
XOR = 0
i = 0
while i < n:
XOR = XOR ^ arr[i]
i += 1
return XOR
Transformation-sequence example
This subsection shows a pair from the stratified subset together with its transformation sequence set. The original program is:
def check_smaller(test_tup1, test_tup2):
return all(x > y for x, y in zip(test_tup1, test_tup2))
The modified program is:
def check_smaller(test_tup1, test_tup2):
if sum((13, 7, 11)) ^ 27 == 0:
cursor_scale = 2 ** 20 >> 10 ^ 42, 'ab'[0] +
chr(48), (3, 5, 7)[1]
try:
pass
except:
pass
return all(y > x for x, y in zip(test_tup1, test_tup2))
Its transformation sequence set is , so . The pair is non-equivalent () because COM (id 4) is altering. COM turns x > y into x < y, and OS (id 8) rewrites x < y as y > x. Applied in the other order, OS and COM produce the same comparison, so both sequences belong to . In T3, both sequences count as correct, and in T2 the correct answer is id 4.
The position of DCI_V2 (id 7) matters for two reasons. First, if DCI_V2 comes before COM, COM also flips the comparison == 0 in the inserted if block to != 0. Second, DCI_V2 chooses the name of the inserted variable from a hash of the program it modifies, so the sequence (4, 7, 8) inserts token_depth instead of cursor_scale. The other orders therefore produce three further modified programs, each forming a separate pair in SCRBench, with transformation sequence sets , , and .
Generation procedure
EXTEND (line 11 of Algorithm 1) takes every valid prefix of the parent collection and appends every transformation of the pool whose id is not already in the prefix. For each new prefix, it checks Definition 1: it compares the ASTs before and after the transformation and runs the tests in on the new program with a time limit of 30 seconds. EXTEND stores every new prefix together with whether it is valid, so the 20 collections also record invalid prefixes. This allows an audit of the generation by the size identity . Here counts the valid prefixes of the parent collection, and is the number of pool transformations not yet in a prefix. If the pool is , then . If the pool is and the parent prefixes already contain preserving transformations, then . A transformation that crashes, or any other silently dropped prefix, violates this identity.
Integrity checks
After the pairs are formed, the pipeline aborts if any of the following checks fails.
Every sequence in uses distinct transformation ids and contains at most one altering transformation.
All sequences in have the same length .
Either every sequence in contains an altering transformation or none does, and this matches the label .
No pair is labeled both equivalent and non-equivalent.
For each length , the sets of all pairs together contain exactly the valid sequences generated at that length, so no sequence is lost or counted twice.
Checks (ii) and (iii) back the definitions of and (Definition 2 and § 2.4). No transformation in the catalog undoes another: renames are never reverted, inserted dead code is never removed, the forwhile rewrite has no inverse, and each operator swap, which is its own inverse, appears at most once in a sequence. We therefore expect no two sequences of different lengths to produce the same pair, and check (ii) confirms this for every pair.
Corpus statistics
Table 6 breaks down the 67,658 labeled pairs of SCRBench by transformation sequence length and label. The non-equivalent pairs at form the smallest stratum and limit the number of pairs drawn per stratum (§ 2.4).
Stratified sampling
The stratified subset draws pairs from each of the ten strata, one for each combination of and , for 4,500 pairs in total.
Coverage target. The presence set of a pair is the set of transformation ids that occur in at least one sequence in . For each stratum, is the set of ids whose coverage the sampler maximizes. For a set of transformations, is the set of their ids. For the equivalent strata (), , because no altering id occurs there. For the stratum (), , because each sequence there is a single altering transformation. For the other non-equivalent strata, .
Maximin cover. The sampler looks for the largest threshold such that some pairs of the stratum contain every id of in the presence sets of at least of them. It binary-searches over and tests each threshold with a randomized greedy set-multicover heuristic, which repeatedly adds the pair that covers the most ids still below the threshold. The heuristic makes up to three attempts per threshold. The sampler keeps the cover found for the largest feasible , fills the remaining slots with pairs drawn uniformly at random without replacement, and shuffles the result.
Seeds and T2 subset. All draws are deterministic: each stratum’s random generator is seeded with a CRC-32 hash of a descriptor that names the stratum and includes a global seed (42), and the final shuffle uses the global seed. The T2 subset consists of the 2,250 non-equivalent pairs of the same draw (450 per ). It is obtained by filtering, not by resampling, so T2 uses the same non-equivalent pairs as T1 and T3.
Verification. A final pass checks the size of every stratum, the coverage threshold achieved in each, and that no equivalent stratum contains an altering id.
Evaluation prompts
This appendix gives the prompt templates for the three tasks and the rule that maps each response to a prediction. We send each prompt as a single user message with no system prompt. We reproduce the templates verbatim, including the Markdown markup that the model receives, such as .... The symbol marks a line wrapped for display
†The T1 Base prompt has no example.
only. In every template, the original program replaces {code_1}, and the modified program replaces {code_2}. In the T2 and T3 templates, the transformation catalog of Appendix B.1 replaces {transformation_description}.
Prompt variants. Each variant fills two slots of a shared template (Table 7). The answer instruction either asks for the answer alone (direct) or asks the model to reason step by step and write the answer on the last line (CoT). The example is either a worked one-shot example — a program pair together with its correct answer — or a format-only example that shows the expected answer line without a program pair. The one-shot example for T1 and T2 is a single altering transformation (id 1, AOM) applied to an MBPP program; the one for T3 applies transformations 5, 7, and 1 in that order. For each task, Appendices B.2–B.4 give the full Base+CoT+one-shot prompt used for the model comparisons in § 3.1, followed by the text that the other three variants replace.
Transformation catalog
The T2 and T3 prompts list all nine transformations of Table 1, one per line, as the id followed by the operational specification below. The T2 prompts list all nine, not only the four altering transformations, so the model must itself determine which transformation altered the semantics.
1: Scope: binary arithmetic operators (+, -, *, /) and
augmented assignments (+=, -=, *=, /=) does not
modify comparisons, logical operators, boolean
literals, or variable names. Procedure: parse to AST;
visit every BinOp and AugAssign node; replace each
operator with its paired opposite (+↔-, *↔/, +=↔-=, *=↔/=,
/↔*); unparses AST back to source. Example: `x = a +
b` → `x = a - b`; `z += 10` → `z -= 10`.
2: Scope: boolean literals `True` and `False` in source
text only; at most one replacement per application
(first match in file order). Does not change
`and`/`or`, comparisons, or non-literal booleans.
Procedure: scan source as text; if `True` appears,
replace the first `True` with `False`; else if `False`
appears, replace the first `False` with `True`;
otherwise leave code unchanged. Example: `if True:
return 1` → `if False: return 1`.
3: Scope: logical boolean operators in BoolOp nodes (and, or) only; does not change arithmetic, comparisons, or True/False literals. Procedure: parse to AST; visit each BoolOp; swap and↔or on that node; unparses AST back to source. Example: if x > 0 and x < 10: → if x > 0 or x < 10:.
4: Scope: comparison operators in Compare nodes: >, <, >=, <=, ==, != (first operator in a compare chain only); does not swap operands or change in/is tests. Procedure: parse to AST; visit Compare nodes; map each operator to its opposite (>↔<, >=↔<=, ==↔!=); unparses AST back to source. Example: if a == b: → if a != b:; while n > 0: → while n < 0:.
5: Scope: user-defined variable and parameter identifiers; renames the single most frequent name (default top_n=1) to generated names VAR_0, VAR_1, . . . that do not appear in the original source. Procedure: parse to AST; count Name and arg occurrences; pick the most common identifier(s); rewrite every load/store/del and matching parameter to VAR_i; unparses AST back to source. Example: result = data + offset → VAR_0 = data + offset when result is the most common name.
6: Scope: inserts one new statement into the deepest statement-owning block (module, function, class, if/for/while/try, etc.); no existing lines are removed or altered. Procedure: parse to AST; locate the deepest body list; insert a standalone Assign to a plausible unused name (e.g. cache_limit) with an opaque constant tuple expression (bit ops, chr, indexing); the name is formed as {word_a}_{word_b} from fixed vocabularies and is guaranteed not to appear anywhere in the source before this transformation runs (no collision with existing identifiers); position at middle of module body or start of nested blocks; unparses AST back to source. Example: adds cache_limit = ((2 ** 20 >> 10) ^ 0x2A), ... inside the deepest block.
7: Scope: same deepest-block targeting as DCI_V1; adds one never-taken if block without modifying reachable code. Procedure: parse to AST; find deepest statement block; insert if with a statically false condition (e.g. (sum((13, 7, 11)) ^ 27) == 0); body holds an unused assign plus an inert try/except with only pass; the dead assign uses the same fresh-name rule as DCI_V1 (guaranteed absent from the original code before transformation); same insert position rules as DCI_V1; unparses AST back to source. Example: adds if (sum((13, 7, 11)) ^ 27) == 0: token_scale = ...; try: pass except: pass inside the deepest block.
8: Scope: Compare nodes where both operands are simple names (ast.Name); operators >, <, >=, <= only (equality == and != are skipped). Procedure: parse to AST; for each qualifying comparison, swap left and right operands and replace the operator with the equivalent flipped form (≥↔≤, >↔<); unparses AST back to source. Example: if a > b: → if b < a:; while low <= high: → while high >= low:.
9: Scope: for loops whose iterator is a direct range(...) call (one, two, or three arguments); other loop forms are left unchanged. Procedure: parse to AST; for each matching For node, emit (1) an assignment initializing the loop variable to the range start, (2) a While with end test (< for positive step, > for negative step), (3) a body equal to the original loop body plus an increment/decrement by the step; unparses AST back to source. Example: for i in range(3): body → i = 0 then while i < 3: body; i += 1.
T1: Detecting semantic equivalence
Base+CoT+one-shot prompt.
You are a Python code expert.
Determine if the following Code1 and Code2 are functionally
equivalent, meaning that they have the same output for
any given input.
Think step by step: compare Code1 and Code2, identify any
differences in logic, control flow, edge-case handling,
or return values, and reason about whether those
differences could produce different outputs for any
input.
After your analysis, write the final answer on the last
line as exactly "YES" or "NO", with no other text on
that line.
---
**Example:**
Input:
Code1:
def first_repeated_char(str1):
for index, c in enumerate(str1):
if str1[:index + 1].count(c) > 1:
return c
Code2:
def first_repeated_char(str1):
for index, c in enumerate(str1):
if str1[:index - 1].count(c) > 1:
return c
The last line of the output:
NO
---
**Task:**
Run the task on the following input:
Code1:
{code_1}
Code2:
{code_2}
Other variants. Base and Base+one-shot replace the two-line CoT instruction (“Think step by step: …”) with the direct instruction:
Respond with exactly "YES" if they are equivalent or "NO"
if they are not.
Do not include any additional text or explanation.
Base+one-shot keeps the one-shot example and labels its answer Output: instead of The last line of the output:. Base and Base+CoT remove everything from the first --- line through Run the task on the following input:. In its place, Base+CoT shows the format-only example Example of the last line of the response: followed by NO; Base shows no example.
Parser. The parser takes the last non-empty line of the response, strips surrounding whitespace, and accepts it only if it equals YES or NO, ignoring case. Any other response is unparsable.
T2: Diagnosing semantic change
Base+CoT+one-shot prompt.
You are a Python code expert.
You are given two code snippets **Code1** and **Code2**.
**Code 2** is produced by applying one or more
transformations on **Code1**.
Exactly one of the applied transformations alters the
semantic of the code snippet, while other applied
transformations (if any) preserve the semantic.
Your task is to determine which transformation was applied
to **Code 1** to alter its semantics and produce **Code
2**.
Think step by step: compare Code 1 and Code 2, list the syntactic differences, and identify which transformation ID best explains the semantic change. After your analysis, write the final answer on the last line as a single digit only (one of 1, 2, 3, 4, 5, 6, 7, 8, 9), with no other text on that line.
Below is a list of all possible transformations, each with an ID and description. Available Transformations: {transformation_description}
Example:
Input:
Code 1:
def first_repeated_char(str1):
for index, c in enumerate(str1):
if str1[:index + 1].count(c) > 1:
return c
Code 2:
def first_repeated_char(str1):
for index, c in enumerate(str1):
if str1[:index - 1].count(c) > 1:
return c
The last line of the output:
1
Task: Run the task on the following input:
Code 1:
{code_1}
Code 2:
{code_2}
Other variants. Base and Base+one-shot replace the CoT instruction (“Think step by step: . . . ”) with the direct instruction:

Figure 1 (PDF p. 16).
Base+one-shot keeps the one-shot example and labels its answer Output: instead of The last line of the output:. Base and Base+CoT remove the one-shot example, from Example: through Run the task on the following input:. Instead, they show a format-only example directly after the answer instruction, before the catalog. They also omit the --- line above the catalog and add one between Code 1: and Code 2:. In Base, this example is Example Response: followed by 4. In Base+CoT, it is Example of the last line of the response: followed by 4.
Parser. The parser extracts the first integer on the last non-empty line of the response; if that line contains none, it falls back to the first integer anywhere in the response. The response is unparseable if no integer is found or the integer is not in . We count a parseable prediction of a preserving id (5–9) as incorrect.
T3: Reconstructing transformation sequences
Base+CoT+one-shot prompt.

Figure 2.
Other variants. Base and Base+one-shot replace the two-line CoT instruction (“Think step by step: . . . ”) with the direct instruction:

Figure 3.
Base+one-shot keeps the one-shot example unchanged. Base and Base+CoT remove the one-shot example, from the --- line above Example: through Run the task on the following input:. Instead, they show a format-only example directly after the answer instruction, before the catalog. In Base, this example is Example Response: followed by 5, 7, 1. In Base+CoT, it is Example of the last line of the response: followed by 5, 7, 1. No variant discloses the transformation sequence length .
Parser. The parser reads the final line of the response and splits it on commas. From each comma-separated field it takes the first integer and keeps it if it is in , preserving order, and drops out-of-range integers. The result is the predicted transformation sequence , which § 3.4 counts as correct if it is one of the sequences in . If no integer is kept, the response is unparseable.
Reproducibility
§ 2 and Appendix § 6 specify the full data generation pipeline, including the generation algorithm (Appendix § 6.§ 4), the integrity checks on the resulting pairs (Appendix § 6.§ 5), and the stratified sampling procedure with its fixed seeds (Appendix § 6.7). Appendix § 7 gives every prompt template verbatim together with the response parsers. § 3.1 gives the inference settings: temperature 0, a 4,000-token budget, and one response per pair. We query all models through the OpenRouter API using the identifiers qwen/qwen3-235b-a22b-2507, openai/gpt-oss-120b, mistralai/mistral-small-24b-instruct-2501, meta-llama/llama-3.1-8b-instruct, and openai/gpt-4.1-nano. The experiments issue 94,500 queries, not counting repeated queries. Each task uses eight combinations of model and prompt variant, namely the four prompt variants on Qwen3 and Base+CoT+one-shot on the other four models, and T1 additionally uses Base+CoT on GPT-4.1-nano. Each combination covers 4,500 pairs in T1 and T3 and 2,250 pairs in T2. We will publicly release the code for data generation and experiments, the full corpus of 67,658 pairs, and all raw model responses upon conference acceptance. Providers can update hosted models over time, and OpenRouter can serve the same model from different providers. Decoding at temperature 0 is also not always deterministic. Re-querying the models can therefore yield slightly different numbers. The released responses will let readers recompute every reported number exactly.
References
- [1]Miltiadis Allamanis and Pengcheng Yin. 2025. Disproving program equivalence with LLMs. arXiv preprint arXiv:2502.18473.arxiv.org/abs/2502.18473
- [2]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.arxiv.org/abs/2108.07732
- [3]Sahar Badihi, Yi Li, and Julia Rubin. 2021. EqBench: A dataset of equivalent and non-equivalent program pairs. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 610–614. IEEE.DOI
- [4]Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T. Devanbu, and Baishakhi Ray. 2022. NatGen: Generative pre-training by “naturalizing” source code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 18–30.arxiv.org/abs/2206.07585
- [5]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.arxiv.org/abs/2107.03374
- [6]Berkeley Churchill, Oded Padon, Rahul Sharma, and Alex Aiken. 2019. Semantic program alignment for equivalence checking. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 1027–1040.DOI
- [7]Google Research. 2021. MBPP: Mostly Basic Python Problems. https://huggingface.co/datasets/google-research-datasets/mbpp. Dataset card; released under the CC BY 4.0 license.
- [8]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
- [9]Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. 2024. CRUXEval: A benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 16568–16621. PMLR.arxiv.org/abs/2401.03065
- [10]Jordan Henkel, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. 2022. Semantic robustness of models of source code. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 526–537. IEEE.DOI
- [11]Rob H ierons, Mark Harman, and Sebastian Danicic. 1999. Using program slicing to assist in the detection of equivalent mutants. Software Testing, Verification and Reliability, 9(4):233–262.
- [12]Ashish Hooda, Mihai Christodorescu, Miltiadis Allamanis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. 2024. Do large code models understand programming concepts? Counterfactual analysis for code predicates. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 18738–18748. PMLR.arxiv.org/abs/2402.05980
- [13]Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649–678.DOI
- [14]Marinos Kintis, Mike Papadakis, Yue Jia, Nicos Malevris, Yves Le Traon, and Mark Harman. 2018. Detecting trivial mutant equivalences via compiler optimisations. IEEE Transactions on Software Engineering, 44(4):308–333.DOI
- [15]Cosimo Laneve, Alvise Spanò, Dalila Rossi, Sabina Rossi, and Michele Bugliesi. 2025. Assessing code understanding in LLMs. In Formal Techniques for Distributed Objects, Components, and Systems (FORTE 2025), volume 15732 of Lecture Notes in Computer Science, pages 202–210. Springer, Cham.arxiv.org/abs/2504.00065
- [16]Changshu Liu, Yang Chen, and Reyhaneh Jabbarvand. 2026. CodeMind: Evaluating large language models for implicit and explicit code execution reasoning. IEEE Transactions on Software Engineering, 52(7):2031–2046.DOI
- [17]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, volume 36, pages 21558–21572.
- [18]Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2014. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE Transactions on Software Engineering, 40(1):23–42.DOI
- [19]Nickil Maveli, Antonio Vergari, and Shay B. Cohen. 2025. What can large language models capture about code functional equivalence? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6880–6918. Association for Computational Linguistics.
- [20]Mistral AI. 2025. Mistral Small 3. https://mistral.ai/news/mistral-small-3.
- [21]Muhammad Rashid Naeem, Tao Lin, Hamad Naeem, and Hailu Liu. 2020. A machine learning approach for classification of equivalent mutants. Journal of Software: Evolution and Process, 32(5):e2238.DOI
- [22]George C. Necula. 2000. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation, pages 83–94.
- [23]Simona Nica and Franz Wotawa. 2012. Using constraints for equivalent mutant detection. In Proceedings of the Seventh Workshop on Model-Based Testing (MBT 2012), volume 86 of Electronic Proceedings in Theoretical Computer Science, pages 1–8.DOI
- [24]A. Jefferson Offutt and W. Michael Craft. 1994. Using compiler optimization techniques to detect equivalent mutants. Software Testing, Verification and Reliability, 4(3):131–154.DOI
- [25]OpenAI. 2025a. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
- [26]OpenAI. 2025b. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/.
- [27]Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg, and Parag Singla. 2026. LLMs are brittle to simple code transformations: Introducing CETBench – a benchmark for code-equivalence checking. In Findings of the Association for Computational Linguistics: ACL 2026, pages 41653–41685. Association for Computational Linguistics.
- [28]Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, volume 1, pages 936–946. IEEE.DOI
- [29]Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: An analysis and survey. In Advances in Computers, volume 112, pages 275–378. Elsevier.DOI
- [30]Samuel Peacock, Lin Deng, Josh Dehlinger, and Suranjan Chakraborty. 2021. Automatic equivalent mutants classification using abstract syntax tree neural networks. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 13–18. IEEE.
- [31]Suzette Person, Matthew B. Dwyer, Sebastian Elbaum, and Corina S. Păsăreanu. 2008. Differential symbolic execution. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, pages 226–237.DOI
- [32]Amir Pnueli, Michael Siegel, and Eli Singerman. 1998. Translation validation. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems, volume 1384 of Lecture Notes in Computer Science, pages 151–166. Springer.
- [33]Zhao Tian, Honglin Shu, Dong Wang, Xuejie Cao, Yasutaka Kamei, and Junjie Chen. 2024. Large language models for equivalent mutant detection: How far are we? In Proceedings of the 33rd ACM SIGSOFT international symposium on software testing and analysis, pages 1733–1745.arxiv.org/abs/2408.01760
