WorksComputer Science
PerfArena: Ranking LLMs by Code Performance
Abstract
Industry now relies on LLMs to write a large share of production code. Although that code is mostly, but not always, correct, correctness alone does not determine whether it is fit for repeated and extensive use, which depends on what the code costs in time, memory and energy. Most code-generation benchmarks nonetheless evaluate correctness in isolation, and most performance-oriented benchmarks cover few languages and rarely measure energy consumption. To address this gap, we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions. PERFARENA executes the solutions of each LLM on a Unix-like host and reports correctness together with latency, peak memory, and energy consumption. Correctness oracles, performance oracles, profilers, and code-generation workflows are pluggable in PERFARENA, enabling the integration of new judging backends and measurement tools without changing the pipeline. We evaluate PERFARENA on two competitions: CLBGComp, 10 compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. Currently, PERFARENA holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across 7 programming languages. Across both competitions, got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory, solving 99.7% of LeetCodeComp and 100% of CLBGComp. All PERFARENA leaderboards and measured solutions are publicly available, providing a continuously updated reference that helps developers choose an LLM per language and metric, instead of relying only on correctness rates or pricing.
Introduction
Large Language Models (LLMs) now generate a substantial and growing share of the world’s code. In 2025, Microsoft reported that AI generated 20–30% of its code [1]. Google reported that AI generated more than 25% of its new code in late 2024 [2], and that this share reached 75% in 2026 [3]. As developers delegate more of their implementation work to LLMs, the efficiency of the shipped software increasingly depends on the efficiency of the code that LLMs produce. Therefore, the criteria that the community uses to judge LLM-generated code grow in importance. In practice, those criteria mostly reduce to functional correctness. For example, HumanEval [4] counts an LLM-generated program as a success if it passes a set of unit tests, using pass@k as a metric. Subsequent benchmarks refine this protocol without changing its objective. In particular, EvalPlus [5] strengthens the test suites to expose solutions that only appear correct. To mitigate training-data contamination, LiveCodeBench continually adds newly published problems [6].
Two factors compound this correctness focus. First, LLM developers optimize and report progress mostly on functional correctness. Second, LLMs learn from human-written code, which rarely prioritizes performance. These factors amplify the performance inefficiencies already present in human-written code. Rua and Ali [7] recently called this feedback loop a performance loop of hell, estimating a density of performance anti-patterns in LLM-generated Android apps relative to human-written equivalents. Several other studies report that LLMs struggle to analyze [8] and optimize [9, 10] software performance.
Existing efficiency-oriented benchmarks evaluate different aspects of the computational performance of LLM-generated code. EffiBench [11] measures latency and peak memory, whereas Mercury [12] and ENAMEL [13] primarily evaluate latency, and EvalPerf [14] counts executed CPU instructions. ENAMEL further introduces the efficiency-aware eff@k metric, which extends pass@k from functional correctness to execution efficiency. Other work broadens this scope. For example, EffiBench-X [15] evaluates latency and peak memory across 6 programming languages: Python, C++, Java, JavaScript, Ruby, and Go. EnergyBench [16] comes closest to a unified view by combining functional correctness with energy-efficiency measurements. However, it focuses on correctness-energy trade-offs and prompt sensitivity rather than on maintaining a comparative efficiency ranking of newly released LLMs. Several empirical studies [10, 17, 18] also measure the energy consumption of LLM-generated code, each with its own task set, hardware platform, and measurement protocol. These differences make their results hard to reproduce, and every new LLM needs a fresh evaluation under each setup. Therefore, efficiency evaluation lacks what correctness evaluation already has: a shared arena that reports functional correctness, latency, peak memory, and energy consumption together for every solution, enabling ranking a newly released LLM against its predecessors without repeating the study.
To bridge this gap, we present PERFARENA2. To the best of our knowledge, PERFARENA is the first arena that ranks LLMs by the efficiency of the correct programs that they produce in terms of 3 performance metrics at once (i.e., latency, peak memory, and energy consumption) and across a wider and more representative set of programming languages than prior benchmarks. Currently, PERFARENA covers 7 of the most used languages on GitHub in 2025 [20]. To achieve that, PERFARENA uses a reproducible protocol for executing and monitoring code, and organizes its evaluation into competitions. Each competition is a named set of coding problems, where each problem has a fixed specification that a solution must satisfy. PERFARENA runs every solution through a correctness oracle, then measures the accepted programs with a performance oracle that reports latency, peak memory, and energy consumption. A performance oracle measures every solution of a competition with the same protocol and the same execution conditions in every language. This uniformity enables PERFARENA to compare solutions from different sources on equal terms, whether they come from an LLM, an AI workflow, or an agentic system.
In this article, we evaluate PERFARENA on two competitions: (1) CLBGComp, 10 compute-intensive benchmarks from Computer Language Benchmarks Game (CLBG) [21], a benchmark suite widely used for cross-language performance evaluation, and (2) LeetCodeComp, our curated selection of 99 LeetCode [22] problems whose accepted solutions differ measurably in latency. For our empirical evaluation, we prompt 13 LLMs of various sizes and families with the same prompt template. We then collect their solutions in 7 programming languages: C++, C#, Java, JavaScript, PHP, Python, and TypeScript.
Across both competitions, gpt-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory. Our results show that correct code is not necessarily efficient code. Although 12 LLMs pass more than 90% of the LeetCodeComp problems, the correct solutions of the slowest LLM take on average 1.6× as long to run as those of gpt-6-astra. On CLBGComp, this gap grows to 3.6×. The fastest LLM also depends on the language. For example, on LeetCodeComp, claude-opus-5-5 leads in Python and PHP, whereas gpt-6-astra leads in the other five languages. However, gpt-6-astra is among the fastest LLMs in every language, and claude-opus-5-5, claude-sonnet-5-5, and glm-5.3-flash are among them in all but one. Beyond avoiding the cheapest LLMs on CLBGComp, paying more does not buy faster code. For example, gpt-6-astra has one of the highest prices per token. However, it costs less than half as much per correct solution as the two most expensive LLMs on LeetCodeComp (claude-fable-5-1 and gpt-5.5), and neither generates faster code.
The leaderboard of PERFARENA, where the generated solutions and their measurements can be downloaded, is live at https://perfarena.net. To enable the replication of our experiments, we have made the source code of PERFARENA publicly available at https://github.com/perfarena/perfarena.
The PERFARENA Pipeline
PERFARENA is an arena that executes code and monitors its performance under a fixed protocol. Figure 1 shows its overall architecture. PERFARENA organizes evaluation into competitions, where each competition is a named set of coding problems. For every problem, PERFARENA fixes a specification that a solution must satisfy. To enter a competition, a source (i.e., an LLM, an AI workflow, or an agentic system) provides a submission. A submission bundles candidate solutions that one source produces for one or more problems of one competition, in one language, together with the LLM, prompt, and configuration that generated them. The standard submission workflow follows five main stages: competition and problem selection (❶, Section 3), code generation (❷–❸, Section 4.1), correctness validation (❹–❺, Section 4.2), execution measurement (❻–❼, Section 5.1), and results analysis (❽, Section 5.2).

Figure 1. The PERFARENA architecture and the flow of one submission through steps 1–8.
PERFARENA enforces two boundaries in its workflow. First, correctness validation strictly precedes measurement, avoiding claims of false efficiency gains by incorrect programs. Second, PERFARENA compares solutions only within a fixed cell (i.e., one problem in one language), which holds the task, workload, toolchain, and execution environment constant. To execute these evaluations, PERFARENA uses two pluggable interfaces: a correctness oracle that assigns a verdict to a program, and a performance oracle that measures latency, peak memory, and energy consumption. PERFARENA implements both interfaces either through the LeetCode judge, which it reaches through LeetCode’s submission Application Programming Interface (API), or through custom harnesses that run in isolated environments. Every submission names exactly one correctness oracle and one performance oracle, allowing PERFARENA to attribute each measurement to a specific judging backend and a specific execution environment. PERFARENA also records full provenance throughout its workflow, including prompts, decoding parameters, LLM digests, execution specifications, and profiling tools.
PERFARENA exposes its pipeline through a Representational State Transfer (REST) API. A campaign is one execution of one or more competitions over a chosen set of LLMs under a fixed measurement profile. Template client scripts in PERFARENA drive a campaign under a chosen interaction workflow (e.g., single-shot generation). The PERFARENA web app then reads the same API, enabling users to browse competitions, leaderboards, and the provenance of any submission. The API never runs a program itself. Instead, it records each submission, resolves its two oracles, and dispatches the work asynchronously. Therefore, a client may submit an attempt and poll for the verdict instead of blocking on a slow judge. PERFARENA stores one measurement row per run, stamped with the execution environment, the energy tool, and a unique run identifier. Therefore, every number on a leaderboard traces back to the hardware and the tool that produced it.
Building Competitions
In PERFARENA, a competition is a set of programming problems. PERFARENA currently hosts two competitions: CLBGComp and LeetCodeComp, each built from a different problem source. We source CLBGComp from the 10 whole programs of the CLBG [21], widely used in prior software performance studies [23, 24, 25, 26]. We adopt those benchmarks, and calibrate each benchmark’s workload size to the measurement host. For LeetCodeComp, we sample 99 problems from a corpus of 1,973 public LeetCode problems [22, 27]. We select those problems whose latency differences separate an efficient solution from an inefficient one, taking the most discriminative problems in each of the 3 LeetCode difficulty levels. Every problem carries its specification (i.e., description, inputs, arguments, data types, and examples) for each programming language. We take each problem statement and its reference solutions from this corpus, and we retrieve its difficulty, its problem categories, and its per-language starter code through LeetCode’s GraphQL API [28]. We also obtain the latency distributions that drive problem selection using the same API. LeetCode builds these distributions from the runtimes of the accepted submissions of its users.
PERFARENA evaluates each problem with two distinct test suites: a comprehensive validation workload to establish functional correctness, and a curated measurement workload for performance analysis. Building a competition involves four steps: selecting problems that distinguish efficient from inefficient code (§ 3.1), curating the validation workload (§ 3.2), curating the measurement workload (§ 4), and translating that measurement workload into all supported languages (§ 4.1).
Problem Selection
CLBGComp and LeetCodeComp differ sharply in their suitability for performance benchmarking. Since CLBGComp problems are computationally intensive by design, they naturally distinguish efficient from inefficient solutions. In contrast, LeetCode contains hundreds of problems, many of which are computationally trivial (e.g., deciding whether a given word is a palindrome). For these lighter problems, execution times across all accepted solutions are practically identical, with variance often smaller than measurement noise or the monitoring overhead itself. Consequently, we filter the LeetCode problem library down to a curated subset of problems that exhibit measurable and distinguishable performance differences. We select this subset in three steps, using the historical latency of the accepted submissions on the LeetCode platform:
Collecting Latency Distributions. For each candidate problem, we submit a Python 3 probe solution drawn from the problem-and-solution corpus [27] to unlock LeetCode’s statistics endpoint. We then query LeetCode’s GraphQL API for the latency distribution of the accepted Python 3 submissions, which LeetCode returns as a histogram of buckets (i.e., discrete time intervals into which submission latencies are grouped). From this distribution, we compute the 10{sup}th
, 50{sup}th, and 95{sup}th` percentiles, which mark a fast, a typical, and a slow submission, respectively.
Filtering Measurable Problems. We retain a problem only if it satisfies three criteria: (1) its median latency is at least 10 ms, which removes problems that run too fast to measure reliably; (2) the gap between its 95{sup}th
and 10{sup}thpercentiles is at least 10 ms, which removes problems where fast and slow solutions barely differ; and (3) its distribution spans at least 15 latency buckets, which removes problems whose latencies fall into so few buckets that their percentiles are coarse and unstable.
Ranking Discriminative Problems. We rank the remaining problems by the ratio of their 95{sup}th
to their 10{sup}thpercentile, which tells how many times slower a slow submission is compared to a fast one. Therefore, a high ratio marks a problem where the choice of algorithm strongly changes latency. Since the ratio cancels any factor that scales all latencies alike, it does not depend on how many test cases a problem runs. In each LeetCode difficulty tier, we keep the 33 problems with the highest ratio.
Overall, this procedure distills the initial 1,973 candidate problems down to the 99 problems that form LeetCodeComp. After we drop the problems that the LeetCode judge cannot time reliably, 886 candidates clear the thresholds. Removing the 22 LeetCode Premium-only problems leaves 864 (151 Easy, 486 Medium, and 227 Hard), from which we keep the top 33 per tier.
Validation Workload
PERFARENA fixes the validation workload of each competition when it builds it. For the current competitions in PERFARENA, we build their validation workloads differently.
CLBGComp
PERFARENA compares the output of CLBGComp programs against reference outputs that the C++ reference programs produce at small arguments. For text outputs (e.g., the DNA sequences that fasta prints), the comparison ignores differences in white space and case. PERFARENA also compares the numbers that n-body and spectral-norm print within an absolute tolerance of . For the binary bitmap of mandelbrot, PERFARENA allows at most 16 differing bytes, because floating-point rounding at the boundary of the set differs slightly between languages.
LeetCodeComp
To build its internal validation workload for LeetCodeComp problems, PERFARENA draws test cases from LeetCodeDataset [29]. When building this test set, we drop cases whose inputs fall outside a problem’s stated bounds or are otherwise malformed, leaving 9,188 test cases over the 99 problems. At run time, PERFARENA runs each solution on the shared, language-independent inputs and compares its output, in a canonical form, against the reference. Each language has a harness that reads the same file of inputs and expected outputs of a problem. This setup enables PERFARENA to validate a solution in any of its languages on all cases without the LeetCode judge. Since several outputs may be correct for some problems, PERFARENA applies an order-insensitive validator that accepts legitimate alternative orderings (e.g., sets or graph paths). In our campaign, the LeetCode judge produced every LeetCodeComp verdict that we report. PERFARENA runs the validation workload when it validates offline, and we use it to check the reference solution of each language (G).
Measurement Workload
The current PERFARENA competitions require different sizing procedures. Each CLBGComp problem has a single argument that sets its workload size, which we calibrate to the measurement host. Each LeetCodeComp problem supports multiple distinct inputs, and our test suites ship several small test cases per problem (total: 9,188, 93 per problem) that we reduce to a representative minimal subset.
CLBGComp
Most CLBGComp benchmarks run as a whole program with a single integer argument that sets the problem size. Only 3 benchmarks (k-nucleotide, regex-redux, and reverse-complement) read their input from standard input instead. For each of these benchmarks, the C++ reference program of fasta generates one shared input file, causing every language implementation to read identical bytes. CLBG defines a canonical argument for each benchmark (e.g., 21 for binary-trees). On our measurement host, these canonical arguments make the slowest languages exceed the time limit of a measurement job. Conversely, a small input makes C++ finish within milliseconds, below the noise floor of the energy profilers. Therefore, we calibrate the argument of each benchmark to the measurement host, constrained by two bounds. C++, the fastest of our measured languages [24], must run long enough to rise above the noise floor of the supported energy profilers: Running Average Power Limit (RAPL) hardware counters [30] and CodeCarbon package-energy estimates [31]. That floor sets a lower bound near 1 second. Since Python is slower than other languages [23, 24], its repeated executions must finish within the 1,200-second limit of one measurement job, setting an upper bound.
The two bounds cannot both be tight, because the fastest and the slowest languages may differ by more than an order of magnitude on one problem. Prior studies report such gaps [23], and we observe them on our host (e.g., the median Python solution of binary-trees runs longer than the median C++ solution). To pick an argument between the two bounds, we time the C++ and the Python programs at candidate arguments. We then choose an argument at which C++ runs for about second, and Python still finishes within the job limit. We map that single argument to every language, so that a cross-language comparison never confounds efficiency with a different workload size.
Any change of measurement host requires repeating this calibration step, because a faster or slower execution environment shifts both bounds. We fix the calibrated argument for the campaign of each benchmark, because measurements taken at a different argument are not comparable. We list the calibrated argument or input size of every benchmark in the appendix.
LeetCodeComp
Selecting a measurement workload for LeetCodeComp requires a more complex procedure. Measuring every test case across every problem, language, and LLM does not scale. To capture a reliable performance footprint, code must run long enough for measurement tools (e.g., energy profilers) to gather multiple samples. Simultaneously, executing tests that trigger identical code paths (e.g., repeating the same branches and instructions) adds overhead without providing new insight. Therefore, for LeetCodeComp, we condense each problem into a small set of test cases that still covers every distinct kind of work that the code performs. We then reuse that set across all languages and LLMs. Since every program must pass a full correctness check before PERFARENA measures it, this reduced set serves strictly as a performance measurement workload rather than a correctness oracle.
Figure 2 illustrates the curation pipeline that we follow for LeetCodeComp. We start from LeetCodeDataset [29], a public dataset of Python LeetCode problems that pairs each problem with functional test cases. We use its test cases for the LeetCodeComp problems (A). We cap each problem at 100 test cases, sampling randomly under a fixed seed when a problem has more (B). To select the measurement subset, we adapt a fingerprinting and clustering procedure from SimPoint [32]. On a bare-metal Intel Linux host that exposes perf [33] counters and RAPL [30] energy readings, we run each sampled test case in a loop for 0.3 seconds, and we record a behavioral fingerprint (C) from six hardware counters: retired instructions, cycles, branches, branch misses, cache references, and cache misses. We also record the energy consumption of each case, which we use to keep the highest-energy case. Applying the cap to the 9,188 cases leaves 8,103 cases across 99 problems. Of these, we profile 8,063, with 40 cases across 5 problems failing in the harness.

Figure 2. Curating the measurement workload for LeetCodeComp.
After profiling, we encode a behavioral fingerprint for each test case, namely a vector of the logarithms of its counter values per call, together with its cache-miss rate, branch-miss rate, and instructions per cycle. We use these fingerprints to reduce each problem’s cases to a small representative set (D–E). To achieve that, we first rescale each counter to zero mean and unit variance. This rescaling prevents a large-magnitude counter (e.g., instruction count) from dominating a smaller one (e.g., branch misses). We then apply Principal Component Analysis (PCA) and keep enough components to preserve 90% of the variation, which turns several correlated counters into a few independent axes. On those axes, we cluster each problem’s cases with -means, choosing the number of clusters by the best silhouette score, a standard measure of how cleanly the clusters separate [34]. Since test cases in the same cluster do the same kind of work, we keep only the test case nearest to the cluster centroid (i.e., the medoid). We also keep the single highest-energy case, because an inefficient solution shows its true cost on the heaviest input, which clustering toward the average might otherwise discard. If a problem has fewer than 5 kept test cases, we add one more at a time, each time picking the case least similar to those already kept (i.e., a maximin choice), until the set size reaches 5. Across all LeetCodeComp problems, this procedure keeps 501/8,063 profiled cases (6.2%), 5 per problem at the median and at most 8.
Since PERFARENA now measures only the few kept cases, it weights them so that their average still reflects the whole problem. To achieve that, PERFARENA assigns every original case to the kept case that it most resembles. The weight of a kept case is then the fraction of the problem’s cases assigned to it. Therefore, a kept case that stands in for several similar cases weighs more than one that stands in for few. All weights for each problem add up to 1. This weighted average over the kept cases approximates the average over all cases, at a fraction of the measurement time. The kept test cases can change the values of a problem. However, PERFARENA compares a solution only with the other solutions of its cell, so a change that all solutions of the cell share cancels. Only a change in the order of two solutions matters, and such a change is rare. On 20 cells that we also measure on all their profiled test cases, the kept test cases reverse which of two solutions is faster for 1.5% of the pairs, and which uses less energy consumption for 0.6%. The highest-energy test case also enters this average, with the share of the cases that it most resembles. We keep this behavior-based set instead of simply the largest inputs, because input size alone does not tell the whole story. Input size correlates with the main fingerprint axis at a median absolute correlation of 0.94 over the problems (mean 0.82), accounting for much of the behavior of a test case, but not all of it. We record the random seeds, the fingerprints, and the kept cases, enabling others to reproduce the exact test case selection.
Translating LeetCodeComp Test Cases
For a given LeetCodeComp problem, every language-specific implementation reads the identical test-case inputs from the same files (F), guaranteeing consistency across languages. Therefore, we reuse the minimized Python test suites for the other programming languages. From each problem’s canonical function signature (i.e., method name, parameter types, and return type), we generate a standalone module per language with three parts: a solution stub, a harness that parses the shared input and runs the function, and a build script that standardizes compilation and execution. The harness handles simple types (e.g., scalars, arrays, and nested lists) directly and uses dedicated parsers for complex structures (e.g., trees, linked lists, and stateful objects). Each implementation folds every result into a checksum, to prevent Dead-Code Elimination (DCE) from a compiler.
To perform cross-language translation from Python, we use Claude Opus 4.8, which was the leading LLM across several public benchmarks when we started developing PERFARENA [35]. We further use this LLM to generate the harness code (i.e., the parsers for complex structures, the runner, and the checksum), the test code of each case whose input needs a language-specific construction (e.g., the operation sequence of a stateful design problem), and a reference solution. The inputs themselves need no cross-language translation, because every language reads them from the same JSON files. Claude generates the test suites one case at a time, which localizes translation errors to single cases. We validate each generated case against the LeetCode judge, using correct solutions [27] alongside failing solutions that we have previously generated using qwen2.5:7b and gemma4:e4b. We exclude these two LLMs from the evaluation, because newer LLMs supersede them. When a generated case produces a verdict that disagrees with the LeetCode judge, we regenerate it with Claude Opus 4.8 until all verdicts agree. The reference solution of each language must also pass every case of the validation workload. The validated cases form the final measurement workload (G).
Generating and Validating Solutions
For each competition, PERFARENA first obtains candidate programs from every LLM (stages 2–3 in Figure 1), and then admits only correct programs to measurement (stages 4–5). The remainder of this section describes how PERFARENA generates code from the selected LLMs (§ 5.1) and then validates the correctness of that generated code (§ 5.2).
Code Generation
PERFARENA supports any LLM served through an OpenAI-compatible REST API [36], a standard interface that covers the major LLM families such as Claude, Gemini, and GPT. To construct a query, PERFARENA uses a pre-defined prompt template that provides the problem specification, the required language, the input and output contract, and any language-specific compilation or interface rules. Every LLM receives the same prompt for a given problem, and only the language-specific part changes across languages. Within each cell, PERFARENA fixes the prompt version and the decoding configuration, drawing the same number of independent samples from each LLM (2–3).
The query generation stage is pluggable. Although Figure 1 shows a single-shot workflow, PERFARENA also supports composing multi-step workflows for richer interactions, including a feedback-driven agent that plans, generates, and revises a solution before committing a final one. All submissions of the campaign that we report in this paper use the single-shot workflow. Therefore, our rankings compare LLMs under the same workflow. PERFARENA records the full generation trace in every case, and ships template client scripts for these workflows, starting with the single-shot one. A client script stores the full response and then extracts the code. LLMs usually return their code inside a Markdown code block, which may carry a language tag (e.g., python). The script takes the first code block that carries the tag of the requested language. If no such block exists, it takes the first code block of any kind. If the response has no code block, the script uses the whole response as the source code. The script does not change the logic of the extracted program. Before a correctness oracle runs a program, it applies only mechanical fixes that its harness needs (e.g., standard Java imports and the PHP open tag on CLBGComp). PERFARENA records programs that still fail to compile or run as failures and never re-runs them.
Each sample, that is, one response of an LLM to one prompt, yields 3 files: the extracted source, the raw response, and a metadata record. The metadata records the provider and LLM, the decoding parameters, the sample and problem identifiers, the language, the prompt, the raw response and any reasoning trace, a timestamp, the generation time, and a hash of this record. PERFARENA carries these identifiers into the validation and measurement records. This trail enables PERFARENA to maintain a strict experimental provenance that ensures LLM updates, prompt revisions, or environment drift are never confused with unexplained measurement noise.
Correctness Validation
To avoid wasting compute and polluting the performance results with invalid solutions, PERFARENA requires every program to pass a correctness oracle before it advances to measurement. For LeetCodeComp, PERFARENA supports two correctness oracles: an external oracle that delegates judging to LeetCode’s rate-limited online judge, and a custom offline oracle that runs the validation workload. Across both competitions, a correctness oracle assigns one of the following verdicts: Accepted, Wrong Answer, Runtime Error, Time Limit Exceeded, Compile Error, or Memory Limit Exceeded. A response in the wrong language, or one that the runner cannot dispatch, also counts as a failure. Only an Accepted program moves forward to the execution measurement pipeline (6–7).
Compiled languages need a build step, which must not count towards latency. Therefore, PERFARENA compiles each program before the measurement campaign and outside all timed regions, using a fixed toolchain and optimization policy recorded in the campaign’s measurement profile. To maintain experimental rigor, PERFARENA never pools measurements across different compiler flag profiles. Any modification to compiler settings creates a distinct profile, which holds the toolchain constant. PERFARENA logs compilation failures separately from execution failures, which distinguishes language-generation errors from runtime defects. PERFARENA still stores the verdict and code of every rejected program. Rather than charging failing code an arbitrary latency or energy consumption penalty, PERFARENA counts a rejected program as a miss in pass@k. This design decouples the ability of an LLM to produce functional code from the execution efficiency of that code.
As PERFARENA fills the leaderboards, we audit the solutions that a correctness oracle rejects. For each rejected solution, we reproduce the verdict and read the output difference to confirm that the rejection reflects a defect in the submitted code and not a limitation of the comparison. This audit extends the CLBGComp comparison beyond an exact match, because a correct whole-program solution may differ from the reference output in ways that the benchmark does not define. In particular, PERFARENA applies the following steps before it compares an output against the reference:
Normalize the formatting that the benchmark leaves unspecified, collapsing repeated blanks, trimming each line end, and ignoring blank separator lines and a missing final newline.
Fold letter case for the benchmarks whose reference output is case-free (e.g.,
reverse-complement).Relax the line order for a benchmark whose output lines are independent and self-labeled, because a parallel solution may emit these lines in a different order (e.g.,
binary-trees).Compare floating-point results under a per-problem numeric tolerance, which keeps a last-digit difference
from summation order from rejecting a correct value (e.g., n-body and spectral-norm).
After these steps, a difference in formatting, letter case, line order, or the last digit of a floating-point value no longer causes a rejection. The oracle gives the verdict Wrong Answer only when the code produces a wrong output. During our campaign, this audit leads to 21 fixes to the comparison mechanism of the correctness oracle and the extraction of code from LLM responses. After manual review of rejected solutions during prototyping, we detect 15 rejected solutions that differ from the reference only in formatting, letter case, or line order.
Measuring and Ranking Efficiency
PERFARENA measures every accepted program (stages 6–7 in Figure 1) and aggregates the measurements into per-competition leaderboards (stage 8). The remainder of this section describes how PERFARENA measures each program (Section 5.1) and then ranks the LLMs (Section 5.2).
Execution Measurement
LeetCodeComp
The LeetCode judge reports latency and peak memory, but no energy consumption. Initially, we planned to derive our rankings from that oracle, because it provides rapid results on an industry-standard platform without a time-consuming local measurement campaign. However, the latency that the LeetCode judge reports is inconsistent across the 5 submissions that we make to LeetCode for each program. Across the 1,268 programs that the judge accepts in all 5 submissions, the ratio between a program’s largest and smallest latency reading has a median of 1.26 (mean 1.29) and reaches 8.5 in the worst case. For peak memory, the same ratio is much smaller (median 1.05, at most 1.07). To compute the PERFARENA leaderboards using consistent measurements, we build a custom performance oracle that reports latency, peak memory, and energy consumption.
To provide reliable and fine-grained measurements, the PERFARENA custom performance oracle executes validated programs on trusted, registered environments (6–7). The oracle may run on any Unix-like host and adopts the isolated measurement methodology established for prior studies on cross-language energy consumption [23, 24]. The oracle runs each program in two passes: one measures latency and energy consumption, and the other measures peak memory. To measure peak memory, PERFARENA reads the maximum resident set size that the kernel records for the process (ru_maxrss), which is the same value that GNU time [37] reports. Measuring peak memory in a separate pass keeps the memory accounting from perturbing the latency and energy consumption profiling. The oracle supports multiple profiling backends (i.e., RAPL [30], CodeCarbon [31], and powermetrics [38]). If the host exposes the on-chip RAPL energy counters, PERFARENA defaults to using them. Otherwise, it falls back to using software power-model estimates.
To prevent external interference, PERFARENA evaluates code on dedicated benchmarking hosts that run no LLM generation workloads. The oracle runs as a persistent local HTTP server that processes jobs sequentially, keeping its own startup, network overhead, and scheduling contention out of the reported numbers. PERFARENA also stores the source code of every solution.
For LeetCodeComp, a single call of a solution on a small test case may last only microseconds. This duration is far shorter than the process startup and compilation around it, whose cost differs sharply between interpreted and compiled languages. Therefore, PERFARENA runs each test case of the measurement workload under a shared per-invocation harness. The harness first calls the solution repeatedly on the same input for 30% of a wall-time budget (1.5 s by default), without counting these warmup calls. The warmup lets caches and Just-in-Time (JIT) compilers settle. The harness then counts the calls that it completes in the remaining 70% of the budget, and records the time of this measured window. Figure 3 shows these phases for one test case. We evaluate the suitability of the warmup share by measuring the LeetCodeComp solutions of gpt-6-astra for 10 random problems in each language with the default budget and with a budget three times larger. With the longer warmup, the latency of one call changes by a median of 0.8%, which we consider negligible in every language except C#, where it falls by a median of 3.1%. The energy tool (e.g., CodeCarbon) wraps the whole harness process. Hence, PERFARENA assigns to the window the share of the total energy consumption that matches the share of the latency of the window. Dividing the time and the energy consumption of the window by the number of calls gives the latency and the energy consumption of one call, which we call the per-invocation latency and energy consumption. To compute one value per problem, PERFARENA then takes the weighted mean over the test cases using the cluster weights of the measurement workload. The latency excludes startup and compilation. The energy consumption excludes them only when it accrues at a roughly constant rate across the phases. The peak memory of a solution is the largest peak memory over its test cases. This peak covers the whole harness process, including the language runtime. A failed measurement never counts as zero. A measurement fails when the program crashes or times out on a test case, or when the energy reading of the test case is missing. PERFARENA leaves those test cases out of the weighted mean of every metric that it lacks. It then renormalizes the weights of the remaining test cases, reporting the measurement when at least one test case succeeds.

Figure 3. The harness process of a LeetCodeComp test case. Only calls in the measured window count toward latency. Warmup and measured window are to scale, whereas startup and teardown are not.
To keep a (JIT) compiler from eliminating calls whose results go unused, the per-invocation harness folds each result into a checksum. The C++ harness additionally passes every argument through an optimization barrier before each call. This barrier prevents the compiler from computing a repeated call on an unchanged input once and reusing its result. The harnesses for Python, JavaScript, TypeScript, PHP, Java, and C# check once whether the solution changes its input. They reuse the input when the solution leaves it unchanged, and they give each call a fresh copy otherwise. The C++ harness builds a fresh copy of the input for every call. Therefore, its measured latency includes the cost of this copy, which is the same for every solution of a test case. For a stateful design problem, every harness rebuilds the object before each replay of the operation sequence.
CLBGComp
For CLBGComp benchmarks, PERFARENA first executes the whole program once and discards this warmup run, bringing the input and the program files into the file cache of the operating system. PERFARENA then executes a configurable number of measured runs, recording latency, energy consumption, and the exit status of each run. A separate run measures peak memory once per solution. Every run starts a new process that has no warmup inside it, unlike the LeetCodeComp harness process. Therefore, the latency and energy consumption of a run include the startup of the language runtime.
Measurement Profile
For both competitions, PERFARENA stamps every run with its measurement profile. A measurement profile records the machine (its processor, core count, and kernel version), the energy tool (e.g., RAPL counters or a CodeCarbon estimate), and the measurement configuration. When the runs of one submission report different energy tools or machines, PERFARENA logs the conflict and leaves the energy tool of the submission unset. The leaderboard shows the machine and the energy tool of every row.
Leaderboard Construction
To rank the LLMs, PERFARENA compares the accepted solutions of every two LLMs on each cell that both solve, using a single sample that each LLM generates per cell, and fits a Bradley-Terry model to these duels. To quantify the gap between two LLMs, PERFARENA divides each value by the value of the reference solution of its cell (i.e., the best correct solution that any LLM in the campaign produces for that cell and metric), and compares these ratios over the cells that both LLMs solve. A cell in which an LLM has no accepted solution gives that LLM no duel, and pass@1 reports these misses separately. The online leaderboard ranks the LLMs with the same approach as our evaluation. The PERFARENA online leaderboard also shows two or more LLMs side by side, with their results on every problem of a chosen language. For submissions that the external LeetCode oracle judges, PERFARENA additionally records the latency and peak memory percentiles that the judge reports against the other accepted submissions on the platform.
Leaderboards update as new submissions arrive, keeping the rankings current instead of frozen at a single point in time [19]. All leaderboards and their provenance records are public, but only we can add results to them, because we generate and measure every solution ourselves. Other researchers may deploy the open-source PERFARENA on their own hosts and run their own leaderboards. Each ranking uses the measurements of one performance oracle. An oracle may pool several machines, but only machines with the same hardware and operating system. An administrator of the platform checks this by hand before adding a machine to the pool. PERFARENA also checks that the median latency and energy consumption of a fixed calibration solution on the new machine fall within 5% of those on the first machine. Since we register a host with other hardware as a separate oracle, no ranking mixes hardware. In our campaign, one host measures every solution.
Evaluation
To show the analyses that PERFARENA enables, we evaluate a range of LLMs on both supported competitions. Our evaluation investigates the following research questions:
RQ1: Which LLM generates code that has the best latency, energy consumption, or peak memory?
RQ2: Which LLMs generate the most performant code when we consider latency, energy consumption, and peak memory together?
RQ3: How does the generation cost of an LLM relate to the performance of its code?
Experimental Setup
Selected LLMs
We evaluate 13 LLMs from 6 providers: OpenAI [39, 40, 41, 42, 43], Anthropic [35, 44, 45, 46], Google [47], DeepSeek [48], xAI [49], and Zhipu [50]. The LLMs span a range of sizes, from flagship LLMs (e.g., gpt-6-astra and claude-opus-5-5) to smaller, faster variants (e.g., gpt-5.4-mini). Table 1 lists the LLMs that we evaluate across 7 programming languages: TypeScript, Python, JavaScript, Java, C#, PHP, and C++. These languages are 7 of the 8 most used languages on GitHub in 2025 [20]. We leave out the remaining one, Shell, because LeetCode accepts Shell only for its shell-scripting problems and CLBG has no Shell programs.
| LLM | Provider | $ per 1M (in/out) | Temp. 0 | |
| gpt-6-astra | OpenAI | 5 / 25 | ✗ |
| gpt-5.6-sol | OpenAI | 4 / 20 | ✗ |
| gpt-5.6-luna | OpenAI | 0.2 / 1.2 | ✗ |
| gpt-5.5 | OpenAI | 5 / 30 | ✗ |
| gpt-5.4-mini | OpenAI | 0.37 / 2.25 | ✗ |
| claude-opus-5-5 | Anthropic | 4 / 20 | ✗ |
| claude-opus-4-8 | Anthropic | 5 / 25 | ✗ |
| claude-sonnet-5-5 | Anthropic | 2 / 10 | ✗ |
| claude-fable-5-1 | Anthropic | 10 / 50 | ✗ |
| gemini-3.8-flash | 0.75 / 3.75 | ✗ | |
| deepseek-v4.1-flash | DeepSeek | 0.44 / 1.32 | ✗ |
| grok-4.6 | xAI | 2 / 6 | ✓ |
| glm-5.3-flash | Zhipu | 0.37 / 2.25 | ✓ |
Table 1. Selected LLMs, their price in USD per 1M tokens, and whether they sample at temperature 0 (✓) or at their default temperature (✗).
Prompt Template and Decoding
To make the LLM the only variable within a cell (i.e., one problem solved in one language), we give every LLM the same prompt template and draw one sample per cell. The template fixes the system role and the required response shape (e.g., function name and parameters), making code extraction more deterministic. Figure 4 shows an example of a prompt instantiated with this template for a concrete LeetCode problem in Python.

Figure 4. The experiment-system-with-shape prompt for the Python problem number-of-days-between-two-dates.
To generate the solutions, we request a temperature of 0 from every API that accepts one, which covers deepseek-v4.1-flash, gemini-3.8-flash, grok-4.6, and glm-5.3-flash. However, the Gemini API ignores this parameter for gemini-3.8-flash [51]. The DeepSeek API also ignores that parameter in the default thinking mode of deepseek-v4.1-flash [52]. The OpenAI and Anthropic APIs accept only the default temperature for their reasoning LLMs [53, 54]. Therefore, only grok-4.6 and glm-5.3-flash generate at temperature 0. Every other LLM uses the default temperature of its API. Since we draw one sample per cell, we do not measure how much the code of an LLM varies between samples.
Machine Configuration
We measure on a single-purpose bare-metal host with an Intel Core i7-7600U processor (2 cores, 4 threads, 2.8 GHz base and 3.9 GHz Turbo Boost frequency) and 16 GB of RAM, which runs Ubuntu 22.04 and exposes RAPL. The host keeps its default frequency settings, with Turbo Boost enabled and the powersave governor of the intel_pstate driver [55]. To emulate a more realistic usage scenario, we do not fix the processor frequency, letting the driver adjust it to the load. We also do not pin a solution to a core.
We read energy consumption through CodeCarbon, which takes the package energy from the RAPL counters and adds a RAM estimate (a constant 10 W on our host). To report only the package energy, we subtract this estimate from every value. The RAPL counters update about once every millisecond [56]. Over the measured window of at least 1.05 s on LeetCodeComp, their update interval adds an error of at most 0.1%. We do not subtract the idle power of the package, which makes up 2–3% of a reading. Subtracting it would change the ratio between the energy consumption of two solutions by at most 0.6% on LeetCodeComp and 1.1% on CLBGComp, less than the margins within which two values tie. Of the 8/735 CLBGComp solutions whose energy consumption we discard because they run for less than 50 ms, 4 are binary-trees solutions that skip the work of the benchmark.
We run one measurement at a time, with the per-invocation harness on LeetCodeComp and with whole-program runs on CLBGComp. We repeat each latency and energy consumption measurement 7 times. At the median coefficient of variation of a solution’s runs (1.3% on LeetCodeComp and 2.8% on CLBGComp), 7 runs keep the 95% confidence interval of the mean within ±1.2% and ±2.6%, respectively. For energy consumption, the median coefficients of variation are 2.8% and 2.7%, which keep the interval within ±2.6% and ±2.5%. For peak memory, we measure once per solution, keeping the serial measurement time bounded. To check that one reading suffices, we run a pilot on our host with gpt-6-astraon 20 randomLeetCodeComp` problems per language, taking 5 peak memory readings of each solution. A single reading differs from the mean by a median of 0.1% in TypeScript, 0.2% in C#, Java, and JavaScript, 0.3% in PHP, 0.5% in Python, and 1% in C++. Only C# and Java show larger outliers, with 95% of their readings within 7% and 5.7% of the mean, respectively, against at most 2% for the other languages.
Computing Ranking
To rank the LLMs, we compare their solutions in pairs, similar to how Chatbot Arena compares chatbot answers [57]. Every cell that two LLMs both solve gives one duel per performance metric. The solution with the lower value wins that duel. Two values that differ by less than the measurement noise tie, giving half a win to each LLM. The noise of a difference between two measurements is times the noise of one measurement. For latency and energy consumption, the noise of one measurement is the 95% confidence interval of its mean. Hence, two latency values tie within 1.7% on LeetCodeComp and 3.7% on CLBGComp, and two energy consumption values tie within 3.7% and 3.5%. For peak memory, the noise of one reading in a language is the 95th percentile of the deviation of a single reading from the mean in the pilot. Hence, two values tie within 0.7% in TypeScript, 1% in PHP, 1.5% in JavaScript, 2.4% in Python, 2.8% in C++, 8.1% in Java, and 9.8% in C#. We use the same margins on CLBGComp. A cell that an LLM fails gives it no duel. Therefore, a failure neither raises nor lowers its performance ranking, and pass@1 reports these failures instead.
We fit a Bradley-Terry model [58] to the duels of each metric, using the MM algorithm [59]. The model gives each LLM one strength. The chance that an LLM wins a duel is its strength divided by the sum of both strengths. From these strengths, we report the win probability of each LLM. The win probability of an LLM is its mean chance of beating another LLM on a cell that both solve. A win probability of 50% marks an average LLM. A win probability of 60% means that the LLM wins 6/10 duels against another LLM on average. To rank LLMs within one language, we fit the model to the duels of that language only. An LLM with fewer than 3 solved problems in the language gets no rank there. With one or two solved problems, a single problem would decide at least half of the duels of the LLM in that language. This rule leaves out no LLM on LeetCodeComp and only 4/91 LLM-language pairs on CLBGComp (gpt-5.4-mini in JavaScript and PHP, and glm-5.3-flash in C# and PHP).
Reference Set
The reference set is a set of LLM-generated solutions, where each solution is the best correct solution for a particular problem, language, and performance metric in our campaign (completed 4 November 2026). We call each solution in this set a reference solution.
A win probability tells how often an LLM wins, allowing us to rank LLMs. However, it does not say by how much the LLMs perform better or worse. Therefore, we also report how far the code of each LLM is from the best code of our campaign. The ratio of a solution divides its value by the value of the reference solution of its cell. A ratio of 1 marks the best solution, and 1.2 means 20% worse. Table 2 reports the geometric mean of these ratios over the cells that an LLM solves as the ratio of that LLM. To compare two LLMs, we divide the value of one LLM by that of the other in every cell that both solve. We then take the geometric mean of these quotients, following standard practice for normalized benchmark results [60]. This comparison does not depend on the reference set. For example, a geometric mean of 1.2 means that the code of the first LLM takes 20% longer to run than the code of the second. We freeze the reference set as a fixed reference for future evaluations. Hence, adding an LLM or approach that produces a better solution does not change the ratios of the LLMs already evaluated.
In both competitions, every LLM produces at least one reference solution. Hence, the reference set does not come only from the LLMs that lead the rankings. For LeetCodeComp, the LLMs supply 2,079 reference solutions across all languages and metrics, from 95 (4.6%) for deepseek-v4.1-flash to 250 (12.0%) for claude-fable-5-1. For CLBGComp, the LLMs supply 210 reference solutions, from 5 (2.4%) for gpt-5.4-mini to 37 (17.6%) for gpt-6-astra. The appendix gives the full per-LLM, per-language breakdown.
The reference set includes no human-written solutions, because we found no human-written set that fits all our problems. For LeetCode problems, the corpus that we use [27] carries human-written solutions only in Python. Additionally, the largest open, multi-language collection [61] records no provenance and covers at most 29/99 problems of LeetCodeComp in most of our languages. For CLBGComp, we do not use the human-written programs of CLBG [21] either, because many of them are tuned to the hardware, operating system, and compilers that they were written for (e.g., x86 SIMD intrinsics). Many of them do not build or run unchanged on our host. Given the lack of human-written solutions, a ratio of 1 marks the best solution of the evaluated LLMs in both competitions.
Statistical Analyses
Our evaluation uses four statistical analyses. We deem a result significant when after Holm’s correction [62] over its family of tests. A family holds the 3 metrics of a competition, the 78 pairs of a metric, or the 4 correlations of one analysis. The appendix describes all tests and results.
Overall Differences
To check whether the LLMs differ at all in a metric, we use a permutation test [63] that we call the LLM shuffle test. The test shuffles, inside each cell, which LLM produced which solution, and compares the spread of the Bradley-Terry strengths before and after the shuffle.
Pairwise Differences
To find which pairs of LLMs differ, we bootstrap the data [64] by drawing 10,000 resamples of the same size with replacement. We then refit the Bradley-Terry model on each, and test the strength gap of each of the 78 pairs of LLMs across the resamples. On both competitions, we resample whole problems, because the cells of one problem in different languages are correlated. On CLBGComp, its 10 problems leave these tests little power. LLMs that no other LLM beats significantly are tied leaders.
Effect of the Language
To check whether the language changes the order of the LLMs, we compare the rankings of every two languages with Kendall’s [65]. A of means that two languages order a fraction of the LLM pairs the same way. A second permutation test, the language shuffle test, compares the mean over the 21 pairs of languages with its value after we shuffle the languages of each problem.
Relation to Other Properties
To relate the rankings to other properties of the LLMs (e.g., pass@1 and cost), we use Spearman’s rank correlation [66].
Evaluation Cost
To show what it takes to add an LLM to Perfarena, we report the cost of our evaluation campaign, including the generation, validation, and measurement of all solutions. Adding an LLM takes one sweep. A sweep submits all problems of a competition in each of its languages. On average, a sweep costs $8.11 and 22 hours of serial generation and measurement for LeetCodeComp, and $4.35 and 5 hours for CLBGComp. The cost of a sweep depends strongly on the LLM, from $0.56 for gpt-5.6-luna to $19.56 for claude-fable-5-1 on LeetCodeComp, and from $0.12 for gpt-5.4-mini to $13.59 for claude-fable-5-1 on CLBGComp. Measurement, not generation, takes most of the sweep time. For LeetCodeComp, the validation and measurement of all solutions take 246 hours, 6.1× the generation time. For CLBGComp, they take 45 hours, 2.26× the generation time. In total, our campaign spends $162 on generation (for 19.7 million response tokens), and takes about 15 days to finish. Table 3 gives the full breakdown of the costs per LLM and competition.
| LLM | pass@1 (%) LC | pass@1 (%) CLBG | Response tokens Per sol. LC | Response tokens Per sol. CLBG | Response tokens Sweep LC | Response tokens Sweep CLBG | Cost (USD) Per sol. LC | Cost (USD) Per sol. CLBG | Cost (USD) Sweep LC | Cost (USD) Sweep CLBG | Time (h) Gen. LC | Time (h) Gen. CLBG | Time (h) Meas. LC | Time (h) Meas. CLBG |
| gpt-5.6-luna | 93.1 | 74.3 | 626 | 3,837 | 404K | 200K | 0.0009 | 0.0047 | 0.56 | 0.25 | 1.6 | 0.7 | 21.3 | 5.8 |
| gpt-5.4-mini | 88.5 | 34.3 | 307 | 1,942 | 188K | 47K | 0.0009 | 0.0049 | 0.57 | 0.12 | 0.5 | 0.1 | 18.4 | 2.5 |
| deepseek-v4.1-flash | 97.4 | 64.3 | 2,323 | 42,035 | 1,568K | 1,892K | 0.0033 | 0.0559 | 2.25 | 2.51 | 3.1 | 2.2 | 19.0 | 9.2 |
| gemini-3.8-flash | 95.8 | 92.9 | 1,190 | 13,235 | 790K | 860K | 0.0049 | 0.0501 | 3.28 | 3.25 | 1.9 | 1.5 | 18.6 | 2.3 |
| claude-sonnet-5-5 | 98.1 | 97.1 | 586 | 1,760 | 398K | 120K | 0.0075 | 0.0193 | 5.11 | 1.31 | 0.9 | 0.3 | 22.1 | 1.2 |
| gpt-6-astra | 99.7 | 100.0 | 321 | 1,458 | 222K | 102K | 0.0109 | 0.0391 | 7.55 | 2.74 | 1.5 | 0.7 | 19.9 | 1.8 |
| gpt-5.6-sol | 98.7 | 98.6 | 474 | 3,020 | 324K | 208K | 0.0118 | 0.0626 | 8.09 | 4.32 | 1.5 | 0.9 | 18.7 | 1.1 |
| claude-opus-4-8 | 96.1 | 78.6 | 368 | 1,306 | 245K | 72K | 0.0134 | 0.0378 | 8.93 | 2.08 | 0.9 | 0.2 | 23.1 | 2.5 |
| glm-5.3-flash | 97.1 | 31.4 | 6,494 | 161,627 | 4,371K | 3,556K | 0.0148 | 0.3642 | 9.98 | 8.01 | 15.0 | 9.0 | 9.6 | 6.0 |
| claude-opus-5-5 | 99.0 | 97.1 | 584 | 2,286 | 401K | 155K | 0.0150 | 0.0491 | 10.27 | 3.34 | 1.3 | 0.4 | 20.5 | 0.9 |
| grok-4.6 | 98.4 | 90.0 | 2,332 | 6,974 | 1,590K | 439K | 0.0156 | 0.0435 | 10.62 | 2.74 | 8.0 | 1.7 | 14.7 | 7.6 |
| gpt-5.5 | 98.4 | 94.3 | 817 | 6,100 | 557K | 403K | 0.0274 | 0.1858 | 18.71 | 12.27 | 2.5 | 1.5 | 18.4 | 1.2 |
| claude-fable-5-1 | 98.0 | 97.1 | 410 | 3,839 | 278K | 261K | 0.0288 | 0.1999 | 19.56 | 13.59 | 1.3 | 0.9 | 21.2 | 3.4 |
| All 13 LLMs | 96.8 | 80.8 | 1,300 | 11,312 | 11,337K | 8,314K | 0.0121 | 0.0769 | 105.49 | 56.53 | 40.1 | 20.1 | 245.6 | 45.4 |
Table 3. Correctness and cost of the sweep of each LLM (one submission per language) on LeetCodeComp (LC) and CLBGComp (CLBG), sorted by the cost in USD per correct solution on LC. Per sol. divides the response tokens or the generation cost by the number of correct solutions, and Sweep gives the total of the sweep. Gen. and Meas. are the generation time and the validation and measurement time of the sweep, where Meas. is the wall time between consecutive solutions, without pauses over two hours, minus the generation time. The last row gives the overall rates and the sums over all sweeps. Bold marks the best value in each column.
Correctness of the Generated Code
PERFARENA measures only correct solutions. We validate the LeetCodeComp solutions using the LeetCode correctness oracle, which runs the test cases of the LeetCode judge. We validate the CLBGComp solutions using the custom correctness oracle of PERFARENA. The pass@1 of an LLM is the share of its cells whose single sample passes the correctness oracle. For LeetCodeComp, pass@1 ranges from 88.5% (gpt-5.4-mini) to 99.7% (gpt-6-astra), and 12 LLMs solve more than 90% of the cells. Hence, correctness barely separates the LLMs on LeetCodeComp. For CLBGComp, pass@1 ranges from 31.4% (glm-5.3-flash) to 100% (gpt-6-astra), and only 7 LLMs solve more than 90% of the cells. Therefore, correctness separates the LLMs on CLBGComp more than on LeetCodeComp.
On LeetCodeComp, an LLM usually solves a problem in every language or in none. Of the 1,287 LLM-problem pairs of LeetCodeComp, 89.7% receive the same verdict in all languages (89.2% accepted in every language and 0.5% rejected in every language, Table 18 in the appendix), with a per-language rejection rate between 2.6% (TypeScript) and 4.2% (C++). On CLBGComp, only 77/130 pairs (59.2%) receive the same verdict in all languages. The two least correct LLMs lose the most on LeetCode Hard problems (gpt-5.4-mini falls from 99% on Easy to 73% on Hard, and gpt-5.6-luna from 99% to 87%). The appendix gives the pass@1 of each LLM and its full breakdown per language and per difficulty.
| Verdict across languages | Pairs | Share |
| Accepted in every language | 1,148 | 89.2% |
| Mixed (accepted in some, rejected in others) | 133 | 10.3% |
| Rejected in every language | 6 | 0.5% |
| Total | 1,287 | 100% |
Table 18. Cross-language correctness consistency on LeetCodeComp. Each LLM-problem pair is classified by its accept or reject verdict across the 7 languages (1,287 pairs attempted in at least two languages).
Correctness also says little about which LLM generates faster code. On LeetCodeComp, pass@1 and the latency win probability correlate positively, but the link does not survive Holm’s correction (, ). Additionally, pass@1 shows no significant link with energy consumption (, ). On CLBGComp, the LLMs with higher pass@1 generate code that consumes less energy (, ), but their link with latency does not survive Holm’s correction (, ).
Best LLM per Performance Metric (RQ1)
Table 2 shows that gpt-6-astra and claude-opus-5-5 generate the fastest code on LeetCodeComp, with win probabilities of 62% and 58%, respectively. No other LLM beats these two significantly in terms of latency. On CLBGComp, gpt-6-astra leads with a 72% win probability, tied with claude-fable-5-1, claude-opus-5-5, glm-5.3-flash, gpt-5.5, gpt-5.6-sol, gemini-3.8-flash, and gpt-5.4-mini. On the cells that an LLM and gpt-6-astra both solve, the code of that LLM takes on average 1.14–1.61× as long to run as that of gpt-6-astra on LeetCodeComp and 1.20–3.61× on CLBGComp. The slowest code comes from grok-4.6 on LeetCodeComp (1.61× as long as that of gpt-6-astra), and from gpt-5.6-luna, claude-opus-4-8, and gpt-5.4-mini on CLBGComp (2.6–3.6×). gpt-5.4-mini is among the tied leaders only because it solves 24/70 cells, too few for the tests to separate it from gpt-6-astra, as for glm-5.3-flash in § 7.7.
| LLM | Latency | Energy | Peak memory | Latency ratio | ||||
| LeetCodeComp | CLBGComp | LeetCodeComp | CLBGComp | LeetCodeComp | CLBGComp | LeetCodeComp | CLBGComp | |
| gpt-6-astra | 62 | 72 | 61 | 72 | 50 | 65 | 1.73 | 1.52 |
| claude-opus-5-5 | 58 | 63 | 58 | 61 | 51 | 43 | 1.97 | 1.76 |
| gpt-5.6-sol | 56 | 59 | 50 | 58 | 50 | 49 | 1.96 | 1.87 |
| claude-sonnet-5-5 | 54 | 44 | 54 | 52 | 50 | 47 | 2.10 | 2.29 |
| claude-fable-5-1 | 53 | 65 | 52 | 63 | 51 | 48 | 2.00 | 1.76 |
| glm-5.3-flash | 53 | 61 | 52 | 58 | 49 | 50 | 2.10 | 2.15 |
| gemini-3.8-flash | 51 | 57 | 56 | 53 | 50 | 50 | 1.94 | 2.14 |
| gpt-5.5 | 49 | 60 | 48 | 60 | 49 | 45 | 2.07 | 1.89 |
| gpt-5.4-mini | 47 | 31 | 47 | 30 | 49 | 44 | 1.90 | 5.02 |
| gpt-5.6-luna | 46 | 25 | 45 | 26 | 50 | 51 | 2.10 | 4.38 |
| deepseek-v4.1-flash | 43 | 39 | 45 | 38 | 50 | 51 | 2.37 | 2.50 |
| claude-opus-4-8 | 39 | 29 | 43 | 31 | 50 | 53 | 2.32 | 3.91 |
| grok-4.6 | 39 | 45 | 40 | 48 | 50 | 54 | 2.72 | 2.82 |
Table 2. Win probability (%) on LeetCodeComp and CLBGComp, sorted by latency win probability on LeetCodeComp. Bold marks tied leaders.
The energy consumption of the code follows its latency closely ( on LeetCodeComp and 0.97 on CLBGComp). On LeetCodeComp, gpt-6-astra, claude-opus-5-5, and gemini-3.8-flash are tied leaders, with win probabilities of 61%, 58%, and 56%, respectively. Of these three, only gemini-3.8-flash is among the tied leaders in every language. On CLBGComp, gpt-6-astra leads with a 72% win probability, tied with claude-opus-5-5, gpt-5.5, glm-5.3-flash, gemini-3.8-flash, claude-sonnet-5-5, grok-4.6, and gpt-5.4-mini. On LeetCodeComp, the energy consumption of one call is its latency times the mean package power of the harness process. This power differs little between solutions. Hence, a solution that runs longer also consumes more energy.
On LeetCodeComp, peak memory does not separate the LLMs. Every LLM wins 49–51% of its duels. Neither the pairwise tests nor the LLM shuffle test find a significant difference (). This lack of separation may come from the language runtime using most of the memory of a LeetCodeComp test case, leaving the differences between solutions within the noise of a single reading. On CLBGComp, gpt-6-astra uses the least peak memory, with a 65% win probability. However, only 5/78 pairs differ, and 7 LLMs tie with gpt-6-astra.
Although the order of the LLMs changes with the language, the tied leaders barely change. On LeetCodeComp, the rankings of two languages () agree less than after we shuffle the languages of each problem (, ). Two LLMs lead in at least one language (claude-opus-5-5 in Python and PHP, and gpt-6-astra in the other five languages). However, gpt-6-astra is among the tied leaders in every language, and claude-opus-5-5, claude-sonnet-5-5, and glm-5.3-flash in all but one. On CLBGComp, 10 problems per language are too few to separate the LLMs within a language. Figure 5 shows how the rank of each LLM changes across languages. On LeetCodeComp, the top and the bottom of the latency ranking change little. For example, gpt-6-astra ranks 1st–5th in every language, claude-opus-5-5 1st–6th, and grok-4.6 11th–13th.
Figure 5. Per-language rank of each LLM by win probability on latency, energy consumption, and peak memory on LeetCodeComp and CLBGComp. Languages go from the most to least used according to GitHub in 2025 [20]. The labels abbreviate claude- as cd- and deepseek- as ds-.
The LLMs in the middle move the most. For example, gemini-3.8-flash ranks 2nd in C++ but 10th in Python, and gpt-5.4-mini 4th but 12th in PHP. On CLBGComp, gpt-6-astra ranks 1st–4th in every language in both latency and energy consumption. With respect to peak memory, the rank of 10 LLMs on LeetCodeComp changes by 9 or more places across languages, as expected when no pair of LLMs differs. Tables 4 to 6 in the appendix give the win probability of every LLM in every language and metric.
The LLM shuffle test confirms that the LLMs differ in latency and energy consumption on both competitions () and in peak memory only on CLBGComp (). The pairwise tests separate 42/78 pairs in latency on LeetCodeComp and 21 on CLBGComp. Table 20 lists every test.
| Metric | Duels | Sig. pairs Cells | Sig. pairs Problems | Leaders | Chance | |||
| LeetCodeComp | ||||||||
| Latency | 661 | < 0.001 | 53/78 | 42/78 | 2 | 0.50 | 0.72 | < 0.001 |
| Energy | 661 | < 0.001 | 52/78 | 39/78 | 3 | 0.48 | 0.71 | < 0.001 |
| Peak memory | 661 | 0.092 | 0/78 | 0/78 | 13 | 0.06 | 0.06 | 0.602 |
| CLBGComp | ||||||||
| Latency | 51 | < 0.001 | 33/78 | 21/78 | 8 | 0.55 | 0.54 | 0.518 |
| Energy | 50 | < 0.001 | 23/78 | 17/78 | 8 | 0.46 | 0.49 | 0.248 |
| Peak memory | 51 | 0.019 | 6/78 | 5/78 | 8 | 0.09 | 0.12 | 0.138 |
Table 20. Statistical tests of the Bradley–Terry rankings. Duels: median number of duels per LLM pair. : LLM shuffle test of whether the LLMs differ at all. Sig. pairs: pairs that differ after Holm’s correction, when the bootstrap resamples cells or whole problems. Leaders: LLMs that no other LLM beats significantly, from the problem bootstrap on both competitions. Language: mean Kendall’s between the per-language rankings, its chance level, and the of the language shuffle test.
On LeetCodeComp, gpt-6-astra and claude-opus-5-5 tie for the fastest code and share the lead in energy consumption with gemini-3.8-flash. On CLBGComp, gpt-6-astra leads. Peak memory barely separates the LLMs on either competition.
Best LLMs Across All Performance Metrics (RQ2)
Since our performance metrics lack a common unit and their priorities vary by deployment (e.g., low latency for cloud services, low energy consumption for mobile devices, or low peak memory for containers), a single composite score would unfairly favor specific use cases. Therefore, to compare the LLMs neutrally, we use Pareto dominance [67]. An LLM dominates another if it is significantly better in at least one metric and significantly worse in none. A difference that the pairwise tests do not find significant counts as a tie. Hence, measurement noise cannot decide dominance. A dominated LLM is inferior under any weighting of the metrics. Hence, the non-dominated LLMs include the best LLM for any deployment, without fixing weights in advance.
Figure 6 sorts LLMs into optimality levels. Level 1 holds the non-dominated LLMs. Level 2 holds the LLMs that only Level 1 dominates, and arrows mark direct dominance.

Figure 6. Pareto optimality levels of the LLMs across all performance metrics. An arrow points from an LLM to each LLM that it directly dominates.
gpt-6-astra dominates 11 other LLMs on LeetCodeComp and 10 on CLBGComp. On LeetCodeComp, claude-opus-5-5 shares Level 1 with gpt-6-astra, because no metric separates them significantly. claude-sonnet-5-5 falls to Level 2, because gpt-6-astra beats it significantly in latency and energy consumption. On CLBGComp, glm-5.3-flash and gemini-3.8-flash share Level 1 with gpt-6-astra, because no metric separates them significantly. glm-5.3-flash solves only 22/70 cells, too few for the tests to separate it from gpt-6-astra. gpt-5.5 falls to Level 2, because gpt-6-astra uses significantly less memory, and no other metric separates them. Figure 6 shows that gpt-6-astra is the only LLM on Level 1 of both competitions, because no other LLM significantly beats it on any metric. The lowest level holds grok-4.6 and claude-opus-4-8 on LeetCodeComp, and claude-opus-4-8, gpt-5.6-luna, and deepseek-v4.1-flash on CLBGComp.
On both LeetCodeComp and CLBGComp, gpt-6-astra is the only LLM that no other LLM dominates.
LLM Cost and Performance (RQ3)
We measure the generation cost of an LLM in two units: response tokens and USD. Both units include the reasoning tokens, which providers bill as output tokens. To relate cost to performance independently of correctness, we divide the cost of a sweep by the number of attempted solutions. To reflect what a developer pays, we also report the cost per correct solution. This cost grows when an LLM fails often. Table 3 lists the cost per correct solution and per sweep of each LLM. The table shows that gpt-6-astra emits few response tokens per correct solution on both competitions (321 on LeetCodeComp and 1,458 on CLBGComp). Only gpt-5.4-mini on LeetCodeComp (307) and claude-opus-4-8 on CLBGComp (1,306) emit fewer. However, its price per token is among the highest. A correct gpt-6-astra solution costs $0.011 on LeetCodeComp and $0.039 on CLBGComp. Both gpt-5.6-luna and gpt-5.4-mini cost 8–12× less per correct solution, but their code takes 1.23× as long to run as that of gpt-6-astra on LeetCodeComp, and 2.76×, 3.61×, respectively, on CLBGComp. Conversely, per correct solution, the three most expensive LLMs cost 2.6× (claude-fable-5-1), 2.5× (gpt-5.5), and 1.4× (grok-4.6) as much as gpt-6-astra on LeetCodeComp and 9.3× (glm-5.3-flash), 5.1× (claude-fable-5-1), and 4.7× (gpt-5.5) on CLBGComp. Their code still runs 1.19–1.61× as long on LeetCodeComp and 1.21–1.29× on CLBGComp.
The cost of an LLM relates to the performance of its code only on CLBGComp. There, the LLMs that cost more per attempted solution in USD generate faster code ( with the latency win probability, ). The four cheapest LLMs (gpt-5.4-mini, gpt-5.6-luna, claude-sonnet-5-5, and claude-opus-4-8) drive this link (without them, , ), because they include the slowest three (gpt-5.6-luna, claude-opus-4-8, and gpt-5.4-mini). We find no significant relation between the token count and latency on either competition, or between either cost and latency on LeetCodeComp (, ). The appendix relates the cost to the language of the solution.
On CLBGComp, the LLMs that cost more per attempted solution generate faster code (), mostly because the four cheapest LLMs generate the slowest code. On LeetCodeComp, cost does not relate to speed. gpt-6-astra costs less than half as much per correct solution than the two most expensive LLMs on each competition, yet their code is not faster.
Threats to Validity
Construct Validity
C1. Latency and energy consumption produce similar LLM rankings ( on LeetCodeComp and 0.97 on CLBGComp). On LeetCodeComp, the per-invocation energy consumption of a solution is its per-invocation latency times the average package power of its harness process. This holds for any reading of the package energy consumption, because energy consumption is power times time. The startup of the language runtime raises this average power slightly (I6). The power varies little between solutions, from 13.1 to 16.0 W for 90% of the LeetCodeComp solutions. Therefore, the per-invocation energy consumption of a solution mostly follows its per-invocation latency. We maintain separate reporting for latency, peak memory, and energy consumption to expose task- and language-level metric disagreements.
C2. For LeetCode problem selection, we rank LeetCode problems by the ratio of their to their latency percentile. A percentile near the timing floor of the LeetCode judge might inflate this ratio. The absolute criteria of the selection (a median of at least 10 ms and a percentile gap of at least 10 ms) limit this bias, but some selected problems may rank high partly because of a small percentile rather than a large gap between fast and slow solutions. Moreover, these distributions come from accepted
Python 3 submissions only. Hence, a problem that separates fast from slow solutions in Python may separate them less in another language. Our measurements show such a gap. In the median cell, the slowest correct solution takes as long as the fastest in Python, but in C#, the language with the smallest spread. Every language still separates the solutions, and the problems with the largest spread in Python tend to have a large spread in the other languages too (–).
C3. A correctness oracle that performs validation only by output comparison cannot tell whether a problem solution does the work that the benchmark intends (e.g., a binary-trees solution that computes its checksum in closed form instead of allocating the trees), or whether it does this work in its own language. Such shortcuts may produce misleading results and win every duel in their cell. Perfārena flags for manual review any accepted solution that runs at least two times faster than the median of the other accepted solutions in its cell, but this flag might miss a shortcut that is only moderately faster. In our campaign, this flag finds five such solutions on CLBGComp. One Python n-body solution of claude-opus-5-5 compiles C code and runs it through a foreign-function interface. Hence, its measurements are not those of Python. It runs faster than the next fastest solution in its cell. We leave it out of every performance ranking and of the reference set, but it still counts as correct for pass@1. The other four are binary-trees solutions, of deepseek-v4.1-flash in PHP, Python, and TypeScript and of gpt-5.5 in Python. They compute the checksums with a formula and never build a tree, and they run – faster than the fastest solution in their cell that builds the trees. We leave all five out of every performance ranking and of the reference set, but they still count as correct for pass@1.
C4. Each language has its own libraries, and each language runtime has its own harness (JavaScript and TypeScript, which both run on Node.js, share one measuring loop). For example, the arbitrary-precision arithmetic available to a pidigits solution differs between languages, and only some harnesses copy the input before every call. Our rankings compare LLMs only within one language, where all LLMs share the same libraries and harness. A comparison of absolute values across languages, however, may partly reflect these differences rather than the generated code.
C5. Our rankings are relative to the evaluated LLMs. The reference set holds only LLM-generated solutions. Hence, a ratio of 1.00 marks the best solution of the evaluated LLMs, not the best known solution.
C6. On LeetCodeComp, PERFARENA measures a weighted average over a few kept test cases instead of the average over all test cases of a problem. On the profiling data that drives the selection, the two averages differ by 4.1% at the median over the problems (mean 9.2%), and by up to 67% for random-pick-index. Our rankings compare the solutions within a cell. Hence, a difference that all solutions of a cell share cancels, and only a change in their order matters. To check whether the kept test cases change this order, we measure 10 problems, drawn at random among those that all LLMs solve in both Python and C++, on all their profiled test cases (53–100 per problem), once for 0.3 s per test case. Both averages come from these same runs, so the check isolates the choice of the kept test cases. The single short run adds noise to both averages, which makes the check conservative. Over the 1,560 duels of these 20 cells, 85% of the duels in latency and 83% in energy consumption keep their outcome, with the tie margins of our rankings. Most other duels change between a tie and a win, and only 1.5% in latency and 0.6% in energy consumption change from a win to a loss. Within a cell, the order of the LLMs by the two averages agrees at a median Kendall’s of 0.79. Hence, a ranking from all test cases could differ from ours in close duels, but it would rarely reverse one. The appendix gives these results per language.
Internal Validity
I1. Other processes, frequency scaling, thermal throttling, and drift of the host add noise to every reading. To limit this noise, the runner measures one solution at a time, refuses to start while any core is more than 60% busy, and repeats each latency and energy consumption measurement seven times. The energy consumption readings also include the idle power of the processor package. We do not subtract it. With nothing running, the package of our host draws a median of 0.43 W over 10 minutes. This idle power makes up a median of 3.0% of the energy consumption of a LeetCodeComp solution and 2.0% of a CLBGComp solution. For 90% of the solutions, it makes up between 2.7% and 3.3% on LeetCodeComp and between 1.8% and 2.9% on CLBGComp. Hence, subtracting it would change the ratio between the energy consumption of two such solutions by at most 0.6% on LeetCodeComp and 1.1% on CLBGComp, below the energy consumption tie margins of 3.7% and 3.5%.
I2. Process startup may dominate a short run, and its cost differs between languages. Every solution in a language pays about the same cost. Hence, this cost affects absolute values more than it affects the ranking of the LLMs.
I3. Calling a solution many times over a single input measures a warm state that a single execution never realistically reaches, and might allow a solution or the compiler to skip work. The Java, C#, and Python harnesses reuse one solution object over consecutive calls. Hence, a solution that caches its results might look much faster (e.g., 31 ns per call in Java against 2.5 s in C++ on one problem). The C++ harness keeps the compiler from moving a call out of the loop, but a JIT compiler may still do so. While every harness folds each result into a checksum, to prevent DCE, this still does not prevent caching. The C++, JavaScript, TypeScript, and PHP harnesses create a new solution object for each call. No harness design that runs all calls in one process removes every such effect. A new object for each call removes the chance for memoization optimization opportunities, but it adds the cost of creating the object to every call. Memoization via static variables might still survive across new objects. To measure the possible effect of allowing Java, C#, and Python to profit from memoization opportunities, we scan the code of all 8,720 accepted LeetCodeComp solutions for state that survives between calls, and we review every hit by hand. 31 solutions in 3 problems keep a memo of their results between calls (16 in Java, 13 in C#, one in Python, and one in PHP). The 2× flag of C3 catches only 13 of them. In the Java and C# cells of one problem, most solutions keep such a memo, so the median of the cell is itself fast. Without these 31 solutions, the latency and energy consumption rankings of the LLMs agree with the original rankings at Kendall’s of 0.97 and 1.00, and the tied leaders do not change. By conducting another experiment where we execute the same solutions with a new object for each call, the 30 Java, C#, and Python solutions run 30× slower (median), and the other solutions of their cells run 1.01× slower. The rankings with these values agree with the original rankings at Kendall’s of 0.97 in latency and 0.97 in energy consumption. The tied leaders and the optimality levels of Figure 6 do not change. Hence, this design choice has negligible impact on the rankings of the LLMs.
I4. A correctness oracle may itself be wrong through stale expected outputs or an order-sensitive comparison. The LeetCode judge validates the LeetCodeComp solutions of our campaign, and our custom oracle validates the CLBGComp solutions. The custom oracle diffs every language against expected outputs from reference implementations, after normalizing the output format and allowing a small numeric tolerance where the output holds floating-point values. However, the custom oracle validates each program at small arguments, while the measured runs at the calibrated argument check only the exit status. Hence, an error that appears only at the larger size (e.g., integer overflow or loss of precision) may go undetected. To check for such errors, we run each of the 735 accepted CLBGComp solutions once at its calibrated argument and compare its output with the output of the C++ reference implementation at the same argument, under the rules of the custom oracle. The outputs of 733 solutions match. The other two, the n-body solutions of grok-4.6 in Java and PHP, start from a wrong position of Jupiter. This error changes the initial energy by , below the tolerance of , but after the 15 million steps of the calibrated argument it changes the final energy by . We leave these two solutions out of every performance ranking and of the reference set, as we do with the shortcut solutions of C3, but they still count as correct for pass@1, because the custom oracle accepts them at its arguments.
I5. At least 11 of the LLMs sample at a nonzero temperature. Hence, a rerun may give them a different solution in some cells and change the outcome of some duels. Each ranking rests on the duels of up to 99 problems in each language, which averages out part of this variation. However, we generate one sample per cell and do not resample these LLMs to measure this variation.
I6. A fixed warmup may end before a JIT compiler has finished optimizing a solution [68]. To check our warmup of 0.45 s, we measure the LeetCodeComp solutions of gpt-6-astra for 10 random problems in each language, with our budget of 1.5 s and with a budget of 4.5 s. The latency of one call changes by a median of 0.8% over all languages, and its median change stays within 1.3% in every language except C#. In C#, it falls by a median of 3.1% and by up to 9.1%. Hence, the C# solutions have not reached their final speed when the measured window starts. To check whether this fall changes the C# ranking, we also measure the C# solutions of all 13 LLMs for the same 10 problems at both budgets, 3 times each. The fall differs between solutions (interquartile range 0.6–7.8%), but the Bradley–Terry rankings of the LLMs at the two budgets agree at Kendall’s τ of 0.90 in latency and 0.97 in energy consumption. No LLM moves by more than one rank in latency or two ranks in energy consumption. Two rankings from single runs agree about as much at different budgets as at the same budget (median τ of 0.82 and 0.83 in latency). Hence, the longer warmup changes the C# ranking no more than the noise between two runs does. The energy consumption of one call falls by a median of 1.9% (4.7% in C#), mostly because a longer budget gives less weight to the startup of the language runtime in the average power (C1). Every solution in a language pays about the same startup. Hence, this shift affects the energy consumption of a language more than the order of its LLMs (I2).
I7. On CLBGComp, the pairwise tests resample only 10 problems, which leaves them little power. Hence, a tied leader on CLBGComp is an LLM that the tests cannot separate from gpt-6-astra, not necessarily one as efficient as gpt-6-astra. For example, gpt-5.4-mini, which solves only 24/70 cells, ties for the lead in latency with a 31% win probability. Resampling cells instead separates more pairs of LLMs (33 instead of 21 of the 78 pairs in latency, 23 instead of 17 in energy consumption, and 6 instead of 5 in peak memory), but it ignores that the cells of one problem are correlated. Under both resamplings, gpt-6-astra stays the only LLM among the tied leaders in every metric on both competitions.
External Validity
E1. Every measurement comes from one hardware and toolchain profile. PERFARENA records the machine and the energy tool with every measurement, and our analysis uses the measurements of one machine and one energy tool only. Hence, a ranking may generalize only to comparable hardware. Prior studies show that the relative efficiency of solutions can change across hardware platforms [18, 69], for example when one solution benefits more than another from more cores, a larger cache, or wider vector units. All LLMs in a ranking run on the same machine. Hence, a different host affects all of them, but not necessarily by the same amount. However, rankings from other hosts may still differ.
E2. The sets of languages and evaluated LLMs are a snapshot, and we add languages, problems, and LLMs to PERFARENA continuously. PERFARENA can store a dated snapshot of its leaderboard with its totals, its top submissions, and a summary per language. Hence, a later reader may compare a published ranking with the state on that date.
E3. The LeetCode problems and the CLBG programs are public, together with many of their fastest solutions. Hence, they are almost certainly in the training data of the evaluated LLMs, which may reproduce a memorized efficient solution rather than derive it. PERFARENA does not detect such reuse. Although all LLMs face the same exposure, the results may overstate the efficiency of the code that an LLM generates for new problems.
Related Work
Benchmarks and Arenas
A benchmark fixes a task set and a scoring protocol, enabling comparable results across LLMs, similar to what MMLU and GSM8K do for knowledge and reasoning [70, 71]. For code generation, HumanEval [4] established a function-level, unit-tested evaluation setting and a binary pass/fail notion of correctness that EvalPlus [5] and the code-generation component of LiveCodeBench [6] retain. While the former strengthens the test suites, the latter continually refreshes the task set with newly released problems. NoFunEval [8] broadens the requirements under test beyond functional correctness to efficiency, security, and maintainability, and finds that LLMs with strong functional results generally falter on these non-functional requirements.
Arenas extend a benchmark with continuous updating and relative ranking. Two arena designs differ in how they judge a submission. Chatbot Arena [57] collects crowdsourced pairwise votes and converts them into a ranking, using human preference as a judge. MathArena [19] instead evaluates LLMs automatically against task-specific ground truth and continually incorporates newly released competitions, using freshness as a defense against data contamination. Copilot Arena [72] brings the Chatbot Arena design to code, ranking LLMs by which code completions developers accept, without executing the code or measuring its performance.
PERFARENA keeps the benchmark’s fixed task set and protocol, but replaces the pass/fail verdict with a measurement. A solution earns its position by the latency, peak memory, and energy consumption once a correctness oracle has accepted it. Similar to MathArena, PERFARENA automatically scores submitted solutions, and updates its ranking as new LLMs are released. Unlike MathArena, PERFARENA draws on long-standing public problems whose efficient solutions are also public. Hence, an LLM may reproduce a memorized solution. Similar to Chatbot Arena, our evaluation ranks LLMs by head-to-head wins with a Bradley-Terry model, where a measurement decides each duel instead of a human vote. Each duel compares two solutions of the same problem in the same language. Hence, each duel holds the task, the workload, and the toolchain constant.
Efficiency of LLM-Generated Code
Empirical studies establish that LLM-generated programs vary widely in efficiency. Coignion et al. [73] compare the latency of LLM-generated solutions. Islam et al. [17], Carpendijk et al. [10], and Apsan et al. [69] measure the energy consumption of LLM-generated solutions, showing that they differ substantially from human-written code in energy consumption and that prompting does not consistently reduce energy consumption. Apsan et al. [69] further show that the relative efficiency of LLM-generated and human-written solutions may change across hardware platforms. Similarly, Solovyeva et al. [18] find substantial variation across LLMs, programming languages, and platforms. In particular, they find that generated solutions sometimes match or outperform efficient human-written solutions, but perform substantially worse in other settings. Since the relative efficiency of solutions may change across hardware, threat E1 discusses our single-host setup. Cursaru et al. [74] measure the energy consumption of Code Llama solutions with a hardware power meter and find that asking for energy-efficient code gives equal or worse efficiency. Vartziotis et al. [75] compare the latency, peak memory, and RAPL-measured energy consumption of code from three AI coding assistants with human solutions, and find the generated code slightly less energy-efficient than optimized human code. Niu et al. [76] measure the latency of generated Python and C++ code, and find that correctness and efficiency are not positively related.
Several benchmarks standardize this efficiency evaluation. For example, EffiBench [11] measures latency and peak memory, while Mercury [12] and ENAMEL [13] evaluate latency efficiency. ENAMEL further introduces the efficiency-aware metric and expert-level reference solutions. On the other hand, EvalPerf [14] scores a solution using executed-instruction counts relative to reference solutions at distinct efficiency levels, avoiding much of the instability of latency at the cost of using a proxy that does not directly measure latency, peak memory, or energy consumption. COFFE [77] also counts CPU instructions, on stress test cases that an LLM generates, while PIE [78] times C++ programs in the gem5 simulator, because the authors measured a false mean speedup of between identical programs on real hardware. ECCO [79] measures the latency and peak memory of Python code edits in a fixed cloud image. Instruction counts and simulation give stable values, but they report neither the energy consumption nor the peak memory of a real machine. Therefore, PERFARENA measures on real hardware and controls the noise through a quiet host, warmup, and repeated runs. Most of these function-level benchmarks evaluate a single language, typically Python. EffiBench-X [15] extends this evaluation to six languages. EnergyBench [16] jointly considers functional correctness and energy efficiency, but organizes its analysis around prompt-induced variation rather than a persistent cross-model efficiency ranking. GSO [9], SWE-Perf [80], and SWE-efficiency [81] take the evaluation to real-world repositories and measure whether LLMs produce correctness-preserving, latency-improving patches for existing projects. GSO spans several languages and reports that LLMs particularly struggle with low-level languages (e.g., C/C++). KernelBench [82] asks LLMs to generate faster GPU kernels and times them against PyTorch. Beyond evaluation, Du et al. [83] show that reinforcement learning with execution feedback may improve the latency and peak memory efficiency of an LLM-generated solution. However, their reported optimization experiments are only Python-based and do not measure energy consumption. EffiLearner [84] feeds latency and peak memory profiles back to the LLM to optimize Python code, and Supersonic [85] trains a small model to generate C/C++ optimizations that the Codeforces judge times.
Our finding that equally correct LLMs differ in efficiency agrees with EvalPerf [14] and COFFE [77], which find that correct code is often inefficient and that larger LLMs are not clearly more efficient. PERFARENA adds energy consumption and multiple languages to this picture. It also shows that the fastest LLM changes with the language, and that, on LeetCodeComp, correctness carries across languages while the order of the LLMs does not. Similar to Niu et al. [76], we find no significant link between correctness and latency. However, on CLBGCComp, the LLMs with higher pass@1 generate code that consumes less energy.
PERFARENA addresses the gaps in prior efficiency benchmarking efforts. For every accepted solution, PERFARENA measures energy consumption alongside latency and peak memory. Our evaluation holds the problem and the programming language constant, because it compares two solutions only within one cell. On LeetCodeComp, PERFARENA selects problems whose accepted solutions differ measurably in latency. Additionally, PERFARENA curates its measurement workload from behaviorally distinct test cases, and records the host, the energy source, and the measurement settings, including the compiler flags, behind every measurement. The resulting workload separates the solutions clearly. In the median cell, the slowest correct solution takes as long as the fastest on LeetCodeComp and on CLBGCComp, while the 7 runs of one solution differ by a median of only and (coefficient of variation). PERFARENA turns these measurements into a continuously updated ranking rather than a fixed snapshot.
Cross-Language Energy Measurement and Workload Selection
Implementations of the same programming problems in different languages may exhibit substantial differences in latency, peak memory, and energy consumption. Pereira et al. [23, 24] measure the latency, peak memory, and energy consumption of CLBG across several languages using the RAPL hardware interface [30]. The resulting ranking separates the fastest and slowest languages by almost two orders of magnitude in energy consumption. For an arena that evaluates LLMs across several languages, this effect may dominate a raw aggregate. This is because an LLM may rank highly simply by succeeding more often in a language whose implementations are relatively efficient. To hold both the problem and the programming language constant, our evaluation compares LLMs only within a fixed cell before aggregating its results. A cell in which an LLM has no accepted solution gives that LLM no duel.
Kempen et al. [25] revisit these measurements with a causal model that separates a language from its implementation, controlling for the specific program implementation, the number of active cores, and memory activity. Under these controls, the authors find that the choice of language implementation has no significant effect on energy consumption beyond latency. PERFARENA does not test this claim. Instead, it reports latency, peak memory, and energy consumption separately for every solution without collapsing them into a single score. This approach allows observers to see the tasks and languages on which the metrics agree and those on which they diverge. In our data, the fastest solution of a cell also uses the least energy consumption in 64% of the cells on LeetCodeComp and 80% on CLBGComp.
To keep a full measurement campaign tractable, PERFARENA reduces each problem to a representative set of measurement cases with a fingerprint-and-cluster procedure adapted from SimPoint [32]. SimPoint characterizes execution intervals by program behavior, clusters similar intervals, and selects representative points from those clusters. PERFARENA extends this idea to cross-language benchmarking by fingerprinting on a single anchor language (i.e., Python), then reuses the same reduced inputs in every other language. The same inputs give every language the same problem instance, namely the same kept test cases on LeetCodeComp and the same argument or input file on CLBGComp. The work that a solution does on them still depends on its algorithm and on the harness of its language (threat C4).
Extending and Reproducing PERFARENA
PERFARENA is an open-source infrastructure with a public instance that lets the community consult its results. Others may deploy their own instance, or extend PERFARENA with new competitions or languages. The platform is not tied to one problem set or to one way of producing code. The correctness oracle, the performance oracle and its profiler, the prompt template, and the code-generation workflow are all pluggable. Therefore, a new judging backend, a different energy tool, or an agent workflow plugs in without changing the rest of the platform.
Our reference set also lowers the cost of comparing against our results. For each problem and language, it records, separately for latency, energy consumption, and peak memory, the best value that any evaluated LLM reaches and the LLM that reaches it. We also version these records. A future study on the same host and energy tool requires only one comparison against this reference set, which costs one sweep per LLM, not a full rerun of the campaign. A study on another host must re-measure all released solutions to rank a new LLM, and only the reference solutions to compute ratios, because measurements from different machines or energy tools are not comparable. To place a new LLM in the ranking, its solutions also duel with the released solutions of the evaluated LLMs, and a refit of the Bradley-Terry model ranks it, similar to how a new LLM enters Chatbot Arena [57]. The win probabilities of the other LLMs may then shift slightly.
We release all code, prompts, generated solutions, per-problem measurements, and rankings. The source code of PERFARENA is at https://github.com/perfarena/perfarena, together with guides to deploy an instance and run a campaign. The PERFARENA leaderboard is live at https://perfarena.net. The code of PERFARENA is under the Apache License 2.0. The measurement framework of Pereira et al. [23] that it extends is under the MIT license. The data that PERFARENA produces (i.e., the generated solutions, the measurements, the rankings, and the prompt templates) is under the Creative Commons Attribution 4.0 license. These licenses do not cover third-party content. The LeetCode problem statements, which the released prompts contain, remain the property of LeetCode. The LeetCodeComp test cases come from LeetCodeDataset [29], which is under the Apache License 2.0. The CLBGComp reference implementations come from CLBG [21], which is under the Revised BSD license. Anyone may consult or download the datasets that PERFARENA produces to check a leaderboard, to train or fine-tune LLMs for efficiency, or to build a new study on top of it. Hence, no result in this paper depends on access to our servers.
Conclusion
PERFARENA is a continuously updated arena that ranks LLMs by the latency, peak memory, and energy consumption of their correct code, under a shared protocol that holds the problem and the language constant in every comparison. Across LeetCodeComp and CLBGComp, gpt-6-astra is the only LLM that no other LLM beats significantly in any metric. Our results show that correct code is not necessarily efficient code. Although 12/13 LLMs solve more than 90% of the LeetCodeComp cells (a problem in one language), the correct solutions of the slowest LLM take on average 1.6× as long to run as those of gpt-6-astra. On the heavier CLBGComp programs, this gap grows to 3.6×. Moreover, we find that on LeetCodeComp, an LLM usually solves a problem in every language or in none, but the fastest LLM changes with the language. On CLBGComp, only 59% of the LLM-problem pairs receive the same verdict in all languages. In practice, developers should compare LLMs per language and per metric instead of relying on pass rates or prices. PERFARENA supplies the measurements to achieve that goal.
Per-language efficiency rankings
The paper ranks the LLMs over all languages of each competition. Here we further report, for each language, the LLMs ranked by their win probability in latency, energy consumption, and peak memory, from a Bradley–Terry model that we fit to the duels of that language only. An LLM with fewer than three solved problems in a language gets no rank there. On CLBGComp, each language holds only ten problems, so most LLMs tie for the lead.
Tables 4 to 6 give the win probability of each LLM in each language on both competitions, in latency, energy consumption, and peak memory. The tables that follow rank the LLMs in each language of both competitions. In each of these ranking tables, subtable (a) gives LeetCodeComp and subtable (b) gives CLBGComp. Bold marks the tied leaders, and a dash marks an LLM with no rank in that language.
| LLM | TS LC | TS CLBG | Py LC | Py CLBG | JS LC | JS CLBG | Java LC | Java CLBG | C# LC | C# CLBG | PHP LC | PHP CLBG | C++ LC | C++ CLBG |
| gpt-6-astra | 62 | 77 | 62 | 61 | 63 | 66 | 62 | 69 | 61 | 69 | 55 | 71 | 67 | 85 |
| claude-opus-5-5 | 58 | 60 | 63 | 68 | 57 | 51 | 53 | 69 | 60 | 65 | 68 | 77 | 51 | 49 |
| gpt-5.6-sol | 55 | 64 | 62 | 55 | 50 | 51 | 57 | 56 | 59 | 60 | 56 | 51 | 51 | 72 |
| claude-sonnet-5-5 | 56 | 35 | 61 | 50 | 55 | 44 | 49 | 42 | 53 | 39 | 59 | 48 | 46 | 51 |
| claude-fable-5-1 | 59 | 59 | 47 | 80 | 57 | 53 | 52 | 77 | 46 | 77 | 56 | 56 | 52 | 50 |
| glm-5.3-flash | 56 | 51 | 51 | 78 | 51 | 69 | 53 | 42 | 54 | – | 55 | – | 49 | 54 |
| gemini-3.8-flash | 52 | 49 | 41 | 56 | 52 | 58 | 56 | 52 | 49 | 64 | 49 | 56 | 54 | 60 |
| gpt-5.5 | 44 | 63 | 51 | 43 | 46 | 70 | 51 | 66 | 52 | 43 | 50 | 60 | 49 | 69 |
| gpt-5.4-mini | 48 | 46 | 40 | 13 | 48 | – | 54 | 34 | 49 | 37 | 41 | – | 52 | 18 |
| gpt-5.6-luna | 38 | 14 | 55 | 46 | 44 | 30 | 48 | 20 | 52 | 39 | 44 | 16 | 43 | 22 |
| deepseek-v4.1-flash | 43 | 51 | 39 | 40 | 49 | 41 | 40 | 29 | 41 | 41 | 44 | 42 | 47 | 37 |
| claude-opus-4-8 | 42 | 25 | 48 | 36 | 39 | 30 | 41 | 38 | 35 | 23 | 29 | 25 | 43 | 32 |
| grok-4.6 | 38 | 57 | 30 | 25 | 39 | 37 | 36 | 55 | 40 | 42 | 44 | 47 | 46 | 50 |
Table 4. Latency win probability (%) on LeetCodeComp (LC) and CLBGComp (CLBG), sorted in a descending order of latency win probability on LC. The highest win probability in each column is in bold. A dash marks an LLM with fewer than 3 solved problems in the corresponding language.
| LLM | TS LC | TS CLBG | Py LC | Py CLBG | JS LC | JS CLBG | Java LC | Java CLBG | C# LC | C# CLBG | PHP LC | PHP CLBG | C++ LC | C++ CLBG |
| gpt-6-astra | 61 | 76 | 64 | 61 | 62 | 66 | 62 | 70 | 64 | 70 | 50 | 73 | 67 | 83 |
| claude-opus-5-5 | 58 | 56 | 63 | 67 | 57 | 50 | 52 | 69 | 53 | 52 | 71 | 77 | 52 | 49 |
| gpt-5.6-sol | 53 | 66 | 49 | 48 | 50 | 51 | 57 | 57 | 45 | 58 | 46 | 52 | 51 | 72 |
| claude-sonnet-5-5 | 56 | 50 | 62 | 54 | 55 | 45 | 48 | 42 | 48 | 66 | 63 | 51 | 46 | 50 |
| claude-fable-5-1 | 57 | 57 | 49 | 79 | 56 | 54 | 51 | 77 | 48 | 57 | 49 | 56 | 51 | 52 |
| glm-5.3-flash | 55 | 51 | 51 | 77 | 51 | 70 | 52 | 40 | 53 | – | 52 | – | 50 | 57 |
| gemini-3.8-flash | 51 | 46 | 57 | 55 | 51 | 57 | 56 | 50 | 63 | 49 | 58 | 56 | 54 | 59 |
| gpt-5.5 | 45 | 64 | 50 | 43 | 46 | 68 | 52 | 66 | 50 | 43 | 46 | 58 | 48 | 64 |
| gpt-5.4-mini | 48 | 41 | 43 | 11 | 48 | – | 53 | 37 | 50 | 39 | 35 | – | 52 | 17 |
| gpt-5.6-luna | 39 | 16 | 47 | 46 | 45 | 28 | 50 | 20 | 38 | 38 | 51 | 21 | 44 | 23 |
| deepseek-v4.1-flash | 43 | 49 | 45 | 40 | 50 | 41 | 40 | 28 | 45 | 35 | 43 | 41 | 46 | 38 |
| claude-opus-4-8 | 44 | 22 | 35 | 41 | 40 | 32 | 40 | 43 | 53 | 30 | 44 | 23 | 44 | 32 |
| grok-4.6 | 39 | 54 | 34 | 27 | 40 | 38 | 36 | 51 | 40 | 62 | 42 | 43 | 46 | 55 |
Table 5. Energy consumption win probability (%) on LeetCodeComp (LC) and CLBGComp (CLBG), sorted in a descending order of latency win probability on LC. The highest win probability in each column is in bold. A dash marks an LLM with fewer than 3 solved problems in the corresponding language.
| LLM | TS LC | TS CLBG | Py LC | Py CLBG | JS LC | JS CLBG | Java LC | Java CLBG | C# LC | C# CLBG | PHP LC | PHP CLBG | C++ LC | C++ CLBG |
| gpt-6-astra | 52 | 77 | 50 | 58 | 50 | 71 | 51 | 60 | 50 | 61 | 50 | 68 | 48 | 56 |
| claude-opus-5-5 | 52 | 45 | 52 | 24 | 49 | 49 | 50 | 46 | 51 | 48 | 55 | 46 | 51 | 37 |
| gpt-5.6-sol | 48 | 50 | 50 | 54 | 48 | 57 | 51 | 46 | 50 | 55 | 50 | 39 | 52 | 42 |
| claude-sonnet-5-5 | 50 | 57 | 52 | 25 | 50 | 51 | 49 | 55 | 50 | 43 | 54 | 43 | 50 | 50 |
| claude-fable-5-1 | 53 | 53 | 50 | 40 | 50 | 46 | 50 | 50 | 50 | 47 | 54 | 44 | 48 | 54 |
| glm-5.3-flash | 48 | 46 | 47 | 41 | 52 | 52 | 44 | 50 | 50 | – | 50 | – | 49 | 45 |
| gemini-3.8-flash | 51 | 52 | 50 | 63 | 52 | 51 | 48 | 44 | 50 | 50 | 49 | 52 | 52 | 38 |
| gpt-5.5 | 49 | 41 | 50 | 55 | 48 | 44 | 50 | 43 | 50 | 47 | 47 | 45 | 49 | 43 |
| gpt-5.4-mini | 47 | 52 | 50 | 41 | 49 | – | 50 | 48 | 49 | 48 | 45 | – | 51 | 36 |
| gpt-5.6-luna | 47 | 25 | 50 | 66 | 49 | 40 | 55 | 56 | 50 | 52 | 47 | 58 | 48 | 65 |
| deepseek-v4.1-flash | 51 | 50 | 50 | 72 | 49 | 30 | 53 | 52 | 49 | 47 | 49 | 37 | 51 | 76 |
| claude-opus-4-8 | 50 | 43 | 48 | 61 | 51 | 61 | 50 | 59 | 50 | 46 | 52 | 58 | 51 | 43 |
| grok-4.6 | 52 | 59 | 51 | 49 | 51 | 48 | 50 | 42 | 50 | 55 | 48 | 60 | 51 | 66 |
Table 6. Peak memory win probability (%) on LeetCodeComp (LC) and CLBGComp (CLBG), sorted in a descending order of latency win probability on LC. The highest win probability in each column is in bold. A dash marks an LLM with fewer than 3 solved problems in the corresponding language.
| Latency (LeetCodeComp) | Energy (LeetCodeComp) | Peak memory (LeetCodeComp) | |||
| 62 gpt-6-astra | 61 gpt-6-astra | 53 claude-fable-5-1 | |||
| 59 claude-fable-5-1 | 58 claude-opus-5-5 | 52 grok-4.6 | |||
| 58 claude-opus-5-5 | 57 claude-fable-5-1 | 52 claude-opus-5-5 | |||
| 56 claude-sonnet-5-5 | 56 claude-sonnet-5-5 | 52 gpt-6-astra | |||
| 56 glm-5.3-flash | 55 glm-5.3-flash | 51 gemini-3.8-flash | |||
| 55 gpt-5.6-sol | 53 gpt-5.6-sol | 51 deepseek-v4.1-flash | |||
| 52 gemini-3.8-flash | 51 gemini-3.8-flash | 50 claude-opus-4-8 | |||
| 48 gpt-5.4-mini | 48 gpt-5.4-mini | 50 claude-sonnet-5-5 | |||
| 44 gpt-5.5 | 45 gpt-5.5 | 49 gpt-5.5 | |||
| 43 deepseek-v4.1-flash | 44 claude-opus-4-8 | 48 gpt-5.6-sol | |||
| 42 claude-opus-4-8 | 43 deepseek-v4.1-flash | 48 glm-5.3-flash | |||
| 38 grok-4.6 | 39 grok-4.6 | 47 gpt-5.4-mini | |||
| 38 gpt-5.6-luna | 39 gpt-5.6-luna | 47 gpt-5.6-luna | |||
| Latency (CLBGComp) | Energy (CLBGComp) | Peak memory (CLBGComp) | |||
| 77 gpt-6-astra | 76 gpt-6-astra | 77 gpt-6-astra | |||
| 64 gpt-5.6-sol | 66 gpt-5.6-sol | 59 grok-4.6 | |||
| 63 gpt-5.5 | 64 gpt-5.5 | 57 claude-sonnet-5-5 | |||
| 60 claude-opus-5-5 | 57 claude-fable-5-1 | 53 claude-fable-5-1 | |||
| 59 claude-fable-5-1 | 56 claude-opus-5-5 | 52 gpt-5.4-mini | |||
| 57 grok-4.6 | 54 grok-4.6 | 52 gemini-3.8-flash | |||
| 51 deepseek-v4.1-flash | 51 glm-5.3-flash | 50 gpt-5.6-sol | |||
| 51 glm-5.3-flash | 50 claude-sonnet-5-5 | 50 deepseek-v4.1-flash | |||
| 49 gemini-3.8-flash | 49 deepseek-v4.1-flash | 46 glm-5.3-flash | |||
| 46 gpt-5.4-mini | 46 gemini-3.8-flash | 45 claude-opus-5-5 | |||
| 35 claude-sonnet-5-5 | 41 gpt-5.4-mini |
Table 7. LLMs on TypeScript, ranked by win probability (%) in latency, energy consumption, and peak memory. (a) LeetCodeComp (b) CLBGComp
| Latency (LeetCodeComp) | Energy (LeetCodeComp) | Peak memory (LeetCodeComp) | |||
| 63 claude-opus-5-5 | 64 gpt-6-astra | 52 claude-opus-5-5 | |||
| 62 gpt-6-astra | 63 claude-opus-5-5 | 52 claude-sonnet-5-5 | |||
| 62 gpt-5.6-sol | 62 claude-sonnet-5-5 | 51 grok-4.6 | |||
| 61 claude-sonnet-5-5 | 57 gemini-3.8-flash | 50 gpt-6-astra | |||
| 55 gpt-5.6-luna | 51 glm-5.3-flash | 50 deepseek-v4.1-flash | |||
| 51 gpt-5.5 | 50 gpt-5.5 | 50 gpt-5.4-mini | |||
| 51 glm-5.3-flash | 50 gpt-5.6-sol | 50 gpt-5.6-luna | |||
| 48 claude-opus-4-8 | 49 claude-fable-5-1 | 50 gemini-3.8-flash | |||
| 47 claude-fable-5-1 | 47 gpt-5.6-luna | 50 gpt-5.5 | |||
| 41 gemini-3.8-flash | 45 deepseek-v4.1-flash | 50 claude-fable-5-1 | |||
| 40 gpt-5.4-mini | 43 gpt-5.4-mini | 48 claude-opus-4-8 | |||
| 39 deepseek-v4.1-flash | 35 claude-opus-4-8 | 47 glm-5.3-flash | |||
| 30 grok-4.6 | 34 grok-4.6 | 47 gpt-5.6-sol | |||
| Latency (CLBGComp) | Energy (CLBGComp) | Peak memory (CLBGComp) | |||
| 80 claude-fable-5-1 | 79 claude-fable-5-1 | 72 deepseek-v4.1-flash | |||
| 78 glm-5.3-flash | 77 glm-5.3-flash | 66 gpt-5.6-luna | |||
| 68 claude-opus-5-5 | 67 claude-opus-5-5 | 63 gemini-3.8-flash | |||
| 61 gpt-6-astra | 61 gpt-6-astra | 61 claude-opus-4-8 | |||
| 56 gemini-3.8-flash | 55 gemini-3.8-flash | 58 gpt-6-astra | |||
| 55 gpt-5.6-sol | 54 claude-sonnet-5-5 | 55 gpt-5.5 | |||
| 50 claude-sonnet-5-5 | 48 gpt-5.6-sol | 54 gpt-5.6-sol | |||
| 46 gpt-5.6-luna | 46 gpt-5.6-luna | 49 grok-4.6 | |||
| 43 gpt-5.5 | 43 gpt-5.5 | 41 gpt-5.4-mini | |||
| 40 deepseek-v4.1-flash | 41 claude-opus-4-8 | 41 glm-5.3-flash | |||
| 36 claude-opus-4-8 | 40 deepseek-v4.1-flash | 40 claude-fable-5-1 | |||
| 25 grok-4.6 | 27 grok-4.6 | 25 claude-sonnet-5-5 | |||
| 13 gpt-5.4-mini | 11 gpt-5.4-mini | 24 claude-opus-5-5 |
Table 8. LLMs on Python, ranked by win probability (%) in latency, energy consumption, and peak memory. (a) LeetCodeComp (b) CLBGComp
| Latency | Energy | Peak memory | Latency | Energy | Peak memory | ||||||
| 63 | gpt-6-astra | 62 | gpt-6-astra | 52 | glm-5.3-flash | 70 | gpt-5.5 | 70 | glm-5.3-flash | 71 | gpt-6-astra |
| 57 | claude-opus-5-5 | 57 | claude-opus-5-5 | 52 | gemini-3.8-flash | 69 | glm-5.3-flash | 68 | gpt-5.5 | 61 | claude-opus-4-8 |
| 57 | claude-fable-5-1 | 56 | claude-fable-5-1 | 51 | claude-opus-4-8 | 66 | gpt-6-astra | 66 | gpt-6-astra | 57 | gpt-5.6-sol |
| 55 | claude-sonnet-5-5 | 55 | claude-sonnet-5-5 | 51 | grok-4.6 | 58 | gemini-3.8-flash | 57 | gemini-3.8-flash | 52 | glm-5.3-flash |
| 52 | gemini-3.8-flash | 51 | glm-5.3-flash | 50 | gpt-6-astra | 53 | claude-fable-5-1 | 54 | claude-fable-5-1 | 51 | gemini-3.8-flash |
| 51 | glm-5.3-flash | 51 | gemini-3.8-flash | 50 | claude-sonnet-5-5 | 51 | gpt-5.6-sol | 51 | gpt-5.6-sol | 51 | claude-sonnet-5-5 |
| 50 | gpt-5.6-sol | 50 | deepseek-v4.1-flash | 50 | claude-fable-5-1 | 51 | claude-opus-5-5 | 50 | claude-opus-5-5 | 49 | claude-opus-5-5 |
| 49 | deepseek-v4.1-flash | 50 | gpt-5.6-sol | 49 | deepseek-v4.1-flash | 44 | claude-sonnet-5-5 | 45 | claude-sonnet-5-5 | 48 | grok-4.6 |
| 48 | gpt-5.4-mini | 48 | gpt-5.4-mini | 49 | claude-opus-5-5 | 41 | deepseek-v4.1-flash | 41 | deepseek-v4.1-flash | 46 | claude-fable-5-1 |
| 46 | gpt-5.5 | 46 | gpt-5.5 | 49 | gpt-5.6-luna | 37 | grok-4.6 | 38 | grok-4.6 | 44 | gpt-5.5 |
| 44 | gpt-5.6-luna | 45 | gpt-5.6-luna | 49 | gpt-5.4-mini | 30 | gpt-5.6-luna | 32 | claude-opus-4-8 | 40 | gpt-5.6-luna |
| 39 | grok-4.6 | 40 | grok-4.6 | 48 | gpt-5.5 | 30 | claude-opus-4-8 | 28 | gpt-5.6-luna | 30 | deepseek-v4.1-flash |
| 39 | claude-opus-4-8 | 40 | claude-opus-4-8 | 48 | gpt-5.6-sol | – | gpt-5.4-mini | – | gpt-5.4-mini | – | gpt-5.4-mini |
| (a) LeetCodeComp | (b) CLBGComp |
Table 9. LLMs on JavaScript, ranked by win probability (%) in latency, energy consumption, and peak memory.
| Latency | Energy | Peak memory | Latency | Energy | Peak memory | ||||||
| 62 | gpt-6-astra | 62 | gpt-6-astra | 55 | gpt-5.6-luna | 77 | claude-fable-5-1 | 77 | claude-fable-5-1 | 60 | gpt-6-astra |
| 57 | gpt-5.6-sol | 57 | gpt-5.6-sol | 53 | deepseek-v4.1-flash | 69 | claude-opus-5-5 | 70 | gpt-6-astra | 59 | claude-opus-4-8 |
| 56 | gemini-3.8-flash | 56 | gemini-3.8-flash | 51 | gpt-6-astra | 69 | gpt-6-astra | 69 | claude-opus-5-5 | 56 | gpt-5.6-luna |
| 54 | gpt-5.4-mini | 53 | gpt-5.4-mini | 51 | gpt-5.6-sol | 66 | gpt-5.5 | 66 | gpt-5.5 | 55 | gpt-5.5 |
| 53 | claude-opus-5-5 | 52 | claude-opus-5-5 | 50 | claude-opus-4-8 | 56 | gpt-5.6-sol | 57 | gpt-5.6-sol | 52 | claude-sonnet-5-5 |
| 53 | glm-5.3-flash | 52 | glm-5.3-flash | 50 | claude-fable-5-1 | 55 | grok-4.6 | 51 | grok-4.6 | 50 | deepseek-v4.1-flash |
| 52 | claude-fable-5-1 | 52 | gpt-5.5 | 50 | grok-4.6 | 52 | gemini-3.8-flash | 50 | gemini-3.8-flash | 50 | claude-fable-5-1 |
| 51 | gpt-5.5 | 51 | claude-fable-5-1 | 50 | claude-opus-5-5 | 47 | glm-5.3-flash | 43 | glm-5.3-flash | 48 | gpt-5.4-mini |
| 49 | claude-sonnet-5-5 | 50 | gpt-5.6-luna | 50 | gpt-5.4-mini | 42 | claude-sonnet-5-5 | 42 | claude-sonnet-5-5 | 46 | claude-opus-5-5 |
| 48 | gpt-5.6-luna | 48 | claude-sonnet-5-5 | 50 | gpt-5.5 | 38 | claude-opus-4-8 | 40 | claude-opus-4-8 | 44 | gemini-3.8-flash |
| 41 | claude-opus-4-8 | 40 | deepseek-v4.1-flash | 49 | claude-sonnet-5-5 | 34 | gpt-5.4-mini | 37 | gpt-5.4-mini | 43 | glm-5.3-flash |
| 40 | deepseek-v4.1-flash | 40 | claude-opus-4-8 | 48 | gemini-3.8-flash | 29 | deepseek-v4.1-flash | 28 | deepseek-v4.1-flash | 44 | gemini-3.8-flash |
| 36 | grok-4.6 | 36 | grok-4.6 | 44 | glm-5.3-flash | 20 | gpt-5.6-luna | 20 | gpt-5.6-luna | 42 | grok-4.6 |
| (a) LeetCodeComp | (b) CLBGComp |
Table 10. LLMs on Java, ranked by win probability (%) in latency, energy consumption, and peak memory.
| Latency | Energy | Peak memory | Latency | Energy | Peak memory | ||||||
| 61 | gpt-6-astra | 64 | gpt-6-astra | 51 | claude-opus-5-5 | 77 | claude-fable-5-1 | 70 | gpt-6-astra | 61 | gpt-6-astra |
| 60 | claude-opus-5-5 | 63 | gemini-3.8-flash | 50 | claude-fable-5-1 | 69 | gpt-6-astra | 66 | claude-sonnet-5-5 | 55 | gpt-5.6-sol |
| 59 | gpt-5.6-sol | 53 | claude-opus-5-5 | 50 | claude-sonnet-5-5 | 65 | claude-opus-5-5 | 62 | grok-4.6 | 55 | grok-4.6 |
| 54 | glm-5.3-flash | 53 | glm-5.3-flash | 50 | claude-opus-4-8 | 64 | gemini-3.8-flash | 58 | gpt-5.6-sol | 52 | gpt-5.6-luna |
| 53 | claude-sonnet-5-5 | 53 | claude-opus-4-8 | 50 | gpt-5.6-luna | 60 | gpt-5.6-sol | 57 | claude-fable-5-1 | 50 | gemini-3.8-flash |
| 52 | gpt-5.6-luna | 50 | gpt-5.5 | 50 | gpt-5.6-sol | 43 | gpt-5.5 | 52 | claude-opus-5-5 | 48 | claude-opus-5-5 |
| 52 | gpt-5.5 | 50 | gpt-5.4-mini | 50 | gpt-6-astra | 42 | grok-4.6 | 49 | gemini-3.8-flash | 48 | gpt-5.4-mini |
| 49 | gemini-3.8-flash | 48 | claude-sonnet-5-5 | 50 | gpt-5.5 | 41 | deepseek-v4.1-flash | 43 | gpt-5.5 | 47 | claude-fable-5-1 |
| 49 | gpt-5.4-mini | 48 | claude-fable-5-1 | 50 | glm-5.3-flash | 39 | gpt-5.6-luna | 39 | gpt-5.4-mini | 46 | deepseek-v4.1-flash |
| 46 | claude-fable-5-1 | 45 | gpt-5.6-sol | 50 | gemini-3.8-flash | 37 | gpt-5.4-mini | 38 | gpt-5.6-luna | 47 | gpt-5.5 |
| 41 | deepseek-v4.1-flash | 45 | deepseek-v4.1-flash | 50 | grok-4.6 | 23 | claude-opus-4-8 | 35 | deepseek-v4.1-flash | 46 | claude-opus-4-8 |
| 40 | grok-4.6 | 40 | grok-4.6 | 49 | gpt-5.4-mini | – | glm-5.3-flash | 30 | claude-opus-4-8 | 43 | claude-sonnet-5-5 |
| 35 | claude-opus-4-8 | 38 | gpt-5.6-luna | 49 | deepseek-v4.1-flash | – | glm-5.3-flash | ||||
| (a) LeetCodeComp | (b) CLBGComp |
Table 11. LLMs on C#, ranked by win probability (%) in latency, energy consumption, and peak memory.
| Latency | Energy | Peak memory | Latency | Energy | Peak memory |
| 68 claude-opus-5-5 | 71 claude-opus-5-5 | 55 claude-opus-5-5 | 77 claude-opus-5-5 | 77 claude-opus-5-5 | 68 gpt-6-astra |
| 59 claude-sonnet-5-5 | 63 claude-sonnet-5-5 | 54 claude-fable-5-1 | 71 gpt-6-astra | 73 gpt-6-astra | 60 grok-4.6 |
| 56 claude-fable-5-1 | 58 gemini-3.8-flash | 54 claude-sonnet-5-5 | 60 gpt-5.5 | 58 gpt-5.5 | 58 gpt-5.6-luna |
| 56 gpt-5.6-sol | 52 glm-5.3-flash | 52 claude-opus-4-8 | 56 gemini-3.8-flash | 56 gemini-3.8-flash | 58 claude-opus-4-8 |
| 55 gpt-6-astra | 51 gpt-5.6-luna | 50 gpt-6-astra | 56 claude-fable-5-1 | 56 claude-fable-5-1 | 52 gemini-3.8-flash |
| 55 glm-5.3-flash | 50 gpt-6-astra | 50 gpt-5.6-sol | 51 gpt-5.6-sol | 52 gpt-5.6-sol | 46 claude-opus-5-5 |
| 50 gpt-5.5 | 49 claude-fable-5-1 | 50 glm-5.3-flash | 48 claude-sonnet-5-5 | 51 claude-sonnet-5-5 | 45 gpt-5.5 |
| 49 gemini-3.8-flash | 46 gpt-5.6-sol | 49 deepseek-v4.1-flash | 47 grok-4.6 | 43 grok-4.6 | 44 claude-fable-5-1 |
| 44 deepseek-v4.1-flash | 46 gpt-5.5 | 49 gemini-3.8-flash | 42 deepseek-v4.1-flash | 41 deepseek-v4.1-flash | 43 claude-sonnet-5-5 |
| 44 gpt-5.6-luna | 44 claude-opus-4-8 | 48 grok-4.6 | 25 claude-opus-4-8 | 23 claude-opus-4-8 | 39 gpt-5.6-sol |
| 44 grok-4.6 | 43 deepseek-v4.1-flash | 47 gpt-5.6-luna | 16 gpt-5.6-luna | 21 gpt-5.6-luna | 37 deepseek-v4.1-flash |
| 41 gpt-5.4-mini | 42 grok-4.6 | 47 gpt-5.5 | – glm-5.3-flash | – glm-5.3-flash | – glm-5.3-flash |
| 29 claude-opus-4-8 | 35 gpt-5.4-mini | 45 gpt-5.4-mini | – gpt-5.4-mini | – gpt-5.4-mini | – gpt-5.4-mini |
Table 12. LLMs on PHP, ranked by win probability (%) in latency, energy consumption, and peak memory. (a) LeetCodeComp (b) CLBGComp
| Latency | Energy | Peak memory | Latency | Energy | Peak memory |
| 67 gpt-6-astra | 67 gpt-6-astra | 52 gpt-5.6-sol | 85 gpt-6-astra | 83 gpt-6-astra | 76 deepseek-v4.1-flash |
| 54 gemini-3.8-flash | 54 gemini-3.8-flash | 52 gemini-3.8-flash | 72 gpt-5.6-sol | 72 gpt-5.6-sol | 66 grok-4.6 |
| 52 claude-fable-5-1 | 52 claude-opus-5-5 | 51 claude-opus-4-8 | 69 gpt-5.5 | 64 gpt-5.5 | 65 gpt-5.6-luna |
| 52 gpt-5.4-mini | 52 gpt-5.4-mini | 51 deepseek-v4.1-flash | 60 gemini-3.8-flash | 59 gemini-3.8-flash | 56 gpt-6-astra |
| 51 gpt-5.6-sol | 51 gpt-5.6-sol | 51 grok-4.6 | 54 glm-5.3-flash | 57 glm-5.3-flash | 54 claude-fable-5-1 |
| 51 claude-opus-5-5 | 51 claude-fable-5-1 | 51 claude-opus-5-5 | 51 claude-sonnet-5-5 | 55 grok-4.6 | 50 claude-sonnet-5-5 |
| 49 glm-5.3-flash | 50 glm-5.3-flash | 51 gpt-5.4-mini | 50 claude-fable-5-1 | 52 claude-fable-5-1 | 45 glm-5.3-flash |
| 49 gpt-5.5 | 48 gpt-5.5 | 50 claude-sonnet-5-5 | 50 claude-opus-5-5 | 49 claude-opus-5-5 | 43 gpt-5.5 |
| 47 deepseek-v4.1-flash | 46 grok-4.6 | 49 glm-5.3-flash | 37 deepseek-v4.1-flash | 38 deepseek-v4.1-flash | 42 gpt-5.6-sol |
| 46 claude-sonnet-5-5 | 46 claude-sonnet-5-5 | 49 gpt-5.5 | 32 claude-opus-4-8 | 32 claude-opus-4-8 | 38 gemini-3.8-flash |
| 46 grok-4.6 | 46 deepseek-v4.1-flash | 48 claude-fable-5-1 | 22 gpt-5.6-luna | 23 gpt-5.6-luna | 37 claude-opus-5-5 |
| 43 gpt-5.6-luna | 44 claude-opus-4-8 | 48 gpt-5.6-luna | 18 gpt-5.4-mini | 17 gpt-5.4-mini | 36 gpt-5.4-mini |
| 43 claude-opus-4-8 | 44 gpt-5.6-luna | 48 gpt-6-astra |
Table 13. LLMs on C++, ranked by win probability (%) in latency, energy consumption, and peak memory. (a) LeetCodeComp (b) CLBGComp
Problem Categories
Table 14 gives the number of LeetCodeComp problems in each LeetCode category that holds at least 9 of them, with one example problem each.
| Category | Problems | Example problem |
| Array | 59 | frog-jump |
| String | 32 | slowest-key |
| Dynamic Programming | 24 | 2-keys-keyboard |
| Hash Table | 22 | word-ladder |
| Math | 18 | clumsy-factorial |
| Sorting | 12 | longest-string-chain |
| Greedy | 11 | di-string-match |
| Bit Manipulation | 10 | stickers-to-spell-word |
| Backtracking | 10 | ambiguous-coordinates |
| Binary Search | 9 | split-array-largest-sum |
| Breadth-First Search | 9 | cousins-in-binary-tree |
| Depth-First Search | 9 | tree-of-coprimes |
Table 14. Category distribution of the 99 LeetCodeComp problems. Since a problem may belong to several categories, the counts sum to more than 99.
Calibrated CLBGComp Workloads
Table 15 lists the workload of every CLBGComp benchmark on our host. Seven benchmarks take a command-line argument. The other three read a fasta output from standard input, so the argument that generates this output sets their workload. With every workload, the C++ program runs for about one second.
| Benchmark | Workload | Meaning |
| binary-trees | 18 | tree depth |
| fannkuch-redux | 11 | permutation length |
| fasta | 12,000,000 | output size |
| mandelbrot | 3,504 | image side in pixels |
| n-body | 15,000,000 | simulation steps |
| pidigits | 10,000 | digits |
| spectral-norm | 4,000 | matrix size |
| k-nucleotide | 2,500,000 | fasta input, about 25 MB |
| regex-redux | 500,000 | fasta input, about 5 MB |
| reverse-complement | 25,000,000 | fasta input, about 254 MB |
Table 15. Calibrated workload of each CLBGComp benchmark on our host.
Check of the Reduced LeetCodeComp Workload
Table 16 splits the check of the reduced LeetCodeComp workload (C6 in § 8.1) by language. Both languages give about the same result. In latency, the weighted average over the kept test cases and the average over all profiled test cases give the same duel outcome for 86.4% of the duels in Python and 84.1% in C++, and at most 1.7% of the duels reverse. Energy consumption agrees less in both languages (85.5% and 80.8%), and at most 0.9% of its duels reverse. Most of the other duels change between a tie and a win. The lowest values of come from cells in which most solutions lie within the tie margin of each other. In the two cells with the lowest latency , Python minimum-number-of-increments-on-subarrays-to-form-a-target-array () and C++ shift-2d-grid (), 46 of the 78 pairs of solutions tie on all profiled test cases, so there mostly orders differences within the noise. In these two cells, 88% and 67% of the duels keep their outcome, and two of the 78 duels reverse in each.
| Language | Metric | Duels | Same | Reversed | |
| Python | Latency | 780 | 86.4% | 1.4% | 0.78 (0.21) |
| Python | Energy | 780 | 85.5% | 0.4% | 0.82 (0.31) |
| C++ | Latency | 780 | 84.1% | 1.7% | 0.79 (0.38) |
| C++ | Energy | 780 | 80.8% | 0.9% | 0.71 (0.33) |
| Both | Latency | 1,560 | 85.3% | 1.5% | 0.79 (0.21) |
| Both | Energy | 1,560 | 83.1% | 0.6% | 0.79 (0.31) |
Table 16. Agreement between the two averages of the workload check, per language, on 10 random LeetCodeComp problems with the solutions of all 13 LLMs. Same: duels with the same outcome under the weighted average over the kept test cases and under the average over all profiled test cases, with the tie margins of our rankings. Reversed: duels in which a win becomes a loss. : Kendall’s between the two orders of the LLMs in a cell, median and minimum (in parentheses) over the cells.
Accept Rate by Difficulty
Table 17 breaks the accept rate down by LeetCode difficulty level for each LLM. CLBGComp has no difficulty levels, so this breakdown applies to LeetCodeComp only.
| LLM | Easy | Med. | Hard |
| gpt-6-astra | 100 | 100 | 99 |
| claude-opus-5-5 | 100 | 100 | 97 |
| gpt-5.6-sol | 98 | 100 | 99 |
| gpt-5.5 | 99 | 100 | 97 |
| grok-4.6 | 100 | 99 | 97 |
| claude-sonnet-5-5 | 99 | 100 | 96 |
| claude-fable-5-1 | 100 | 100 | 94 |
| deepseek-v4.1-flash | 99 | 98 | 95 |
| glm-5.3-flash | 100 | 99 | 92 |
| claude-opus-4-8 | 99 | 99 | 91 |
| gemini-3.8-flash | 99 | 98 | 90 |
| gpt-5.6-luna | 99 | 94 | 87 |
| gpt-5.4-mini | 99 | 94 | 73 |
Table 17. Accept rate by LeetCode difficulty on LeetCodeComp (percent), sorted by pass@1. The set holds 33 problems at each level, each measured in the 7 languages.
The set holds 33 problems at each level. Eleven of the 13 LLMs stay at or above 90% at every level. The two weakest fall on the Hard problems, gpt-5.4-mini from 99% on Easy to 73% on Hard and gpt-5.6-luna from 99% to 87%. The same two LLMs also show blind spots in some problem categories. gpt-5.4-mini accepts only 24% of the three Number Theory problems and 69% of the five Memoization problems, and gpt-5.6-luna accepts 43% of the Number Theory problems. The other LLMs score higher in both categories. Number Theory has the lowest accept rate of all categories for every LLM (tied with Linked List for gpt-5.6-sol), and its rate ranges from 24% for gpt-5.4-mini to 90% for gpt-6-astra. Each of these categories holds only three to five problems, so we report them as observations rather than established results.
Cross-Language Correctness Consistency
Table 18 gives the full breakdown behind the cross-language correctness analysis in the main text. Each LLM-problem pair is classified by how the accept or reject verdict varies across the 7 languages the LLM attempts.
The 6 pairs rejected in every language are the only cases where an LLM solves a problem in none of the 7 languages. Five of them belong to gpt-5.4-mini and one to gpt-5.6-luna, and all fall on Hard problems, such as abbreviating-the-product-of-a-range. A mixed pair shows that the LLM can solve the problem but fails in some languages. Such a failure may come from a language-specific coding mistake, from a timeout in a slower language, or from sampling, because each cell holds one sample.
Reference-Set Contributions
Table 19 gives the full per-LLM, per-language breakdown of the reference set summarized in the main text. Each cell contributes its best correct solution for latency, energy consumption, and peak memory separately, so an LLM may win any metric of a cell independently. Each count is the number of these reference values the LLM supplies.
| Panel | LLM | C++ | C# | Java | JS | PHP | Py | TS | Total |
| (a) LeetCodeComp | claude-fable-5-1 | 60 | 19 | 23 | 30 | 57 | 30 | 31 | 250 |
| (a) LeetCodeComp | gpt-6-astra | 31 | 30 | 39 | 46 | 17 | 43 | 41 | 247 |
| (a) LeetCodeComp | claude-opus-5-5 | 22 | 31 | 22 | 30 | 67 | 44 | 26 | 242 |
| (a) LeetCodeComp | gpt-5.5-sonnet-5-5 | 17 | 28 | 24 | 22 | 35 | 39 | 25 | 190 |
| (a) LeetCodeComp | gemini-3.8-flash | 29 | 37 | 24 | 28 | 17 | 18 | 20 | 173 |
| (a) LeetCodeComp | claude-opus-4-8 | 30 | 41 | 23 | 14 | 17 | 14 | 19 | 158 |
| (a) LeetCodeComp | gpt-5.6-sol | 21 | 13 | 24 | 16 | 15 | 24 | 30 | 143 |
| (a) LeetCodeComp | glm-5.3-flash | 23 | 21 | 15 | 27 | 18 | 8 | 21 | 133 |
| (a) LeetCodeComp | gpt-5.6-luna | 9 | 19 | 29 | 20 | 15 | 29 | 10 | 131 |
| (a) LeetCodeComp | gpt-5.4-mini | 14 | 14 | 25 | 20 | 8 | 9 | 19 | 109 |
| (a) LeetCodeComp | gpt-5.5 | 10 | 24 | 14 | 19 | 11 | 13 | 14 | 105 |
| (a) LeetCodeComp | grok-4.6 | 19 | 7 | 17 | 12 | 13 | 12 | 23 | 103 |
| (a) LeetCodeComp | deepseek-v4.1-flash | 12 | 13 | 18 | 13 | 7 | 14 | 18 | 95 |
| (a) LeetCodeComp | Total | 297 | 297 | 297 | 297 | 297 | 297 | 297 | 2079 |
| (b) CLBGComp | gpt-6-astra | 8 | 5 | 0 | 4 | 4 | 6 | 10 | 37 |
| (b) CLBGComp | gpt-5.5 | 1 | 2 | 4 | 6 | 5 | 3 | 4 | 25 |
| (b) CLBGComp | claude-fable-5-1 | 2 | 3 | 7 | 5 | 0 | 5 | 2 | 24 |
| (b) CLBGComp | claude-opus-5-5 | 1 | 5 | 0 | 3 | 9 | 4 | 2 | 24 |
| (b) CLBGComp | gemini-3.8-flash | 3 | 5 | 2 | 3 | 1 | 3 | 0 | 17 |
| (b) CLBGComp | grok-4.6 | 2 | 2 | 6 | 0 | 3 | 0 | 4 | 17 |
| (b) CLBGComp | deepseek-v4.1-flash | 2 | 5 | 3 | 0 | 1 | 0 | 2 | 13 |
| (b) CLBGComp | gpt-5.6-sol | 4 | 3 | 1 | 0 | 2 | 2 | 2 | 13 |
| (b) CLBGComp | glm-5.3-flash | 1 | 0 | 0 | 1 | 2 | 4 | 2 | 10 |
| (b) CLBGComp | claude-opus-4-8 | 0 | 4 | 1 | 1 | 3 | 1 | 0 | 9 |
| (b) CLBGComp | gpt-5.6-luna | 3 | 0 | 1 | 4 | 0 | 1 | 0 | 9 |
| (b) CLBGComp | claude-sonnet-5-5 | 3 | 0 | 1 | 1 | 0 | 0 | 1 | 7 |
| (b) CLBGComp | gpt-5.4-mini | 0 | 0 | 1 | 2 | 0 | 1 | 1 | 5 |
| (b) CLBGComp | Total | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 210 |
Table 19. Reference solutions contributed by each LLM, by language. A cell is one problem in one language, and each cell contributes its best correct solution for latency, energy consumption, and peak memory separately, so an LLM may win any metric of a cell independently. Sorted by total.
Statistical Tests
Section 6.1.4 defines the duels and the Bradley–Terry model, and Table 20 lists the results of the tests that we describe here. The LLM shuffle test checks whether the LLMs differ at all in a metric. Its statistic is the standard deviation of the fitted strengths. If all LLMs were equally efficient, the solutions in a cell would be interchangeable. We therefore shuffle, inside each cell, which of the LLMs that solved the cell produced which solution, and we refit the model 1,000 times. The -value is the share of shuffles whose strengths spread at least as much as the observed ones.
To find which pairs differ, we draw 10,000 bootstrap samples [64] and refit the model on each sample. For each of the pairs, the standard deviation of the strength gap over the samples gives a standard error, and a two-sided test of the observed gap gives a -value. Holm’s procedure [62] corrects the 78 -values of a metric, so that the chance of reporting even one false difference stays below 5%. The cells of one problem in different languages are not independent, so on both competitions the bootstrap resamples whole problems. Table 20 reports both resamplings for both competitions. On LeetCodeComp, resampling cells instead of problems finds more significant pairs (53 instead of 42 in latency and 52 instead of 39 in energy consumption) and drops gemini-3.8-flash from the tied leaders in energy consumption. On CLBGComp, resampling cells finds 33 instead of 21 pairs in latency, 23 instead of 17 in energy consumption, and 6 instead of 5 in peak memory (I7 in § 9).
The language shuffle test checks whether the language changes the order of the LLMs. It compares the per-language rankings with Kendall’s . Two languages would disagree somewhat even if the language did not matter, because each per-language ranking rests on fewer duels. We therefore shuffle, inside each problem, which language each measurement belongs to, recompute the mean 1,000 times, and report as the share of shuffles whose is at most the observed one. The mean over the shuffles is the chance level, and a below 0.05 means that two languages agree less than chance allows.
The tie margin barely changes the rankings. With no margin, so that every difference decides a duel, the latency and energy consumption rankings agree with ours at Kendall’s on both competitions. Only the peak memory rankings change ( on LeetCodeComp and 0.62 on CLBGComp), because many peak memory differences lie within the noise.
A ranking by the geometric mean of the ratios to the reference set, taken over the cells that each LLM solved, would favor the LLMs that fail hard problems. By that measure, gpt-5.4-mini ranks second in latency on LeetCodeComp, because it fails 80 cells, 62 of them on Hard problems. By win probability, it ranks ninth, because each of its duels compares it with another LLM on a cell that both solve.
Additional Analyses
Energy consumption diverges from latency in magnitude on CLBGComp but not on LeetCodeComp. Here, a ratio is the ratio to the reference set (§ 7.2), and we take its geometric mean over the cells that an LLM solves. On LeetCodeComp, the two ratios stay within 2% for every LLM. On CLBGComp, every LLM’s energy consumption ratio is below its latency ratio (13 of 13 LLMs, signed test , the probability of 13 such LLMs out of 13 if a higher and a lower energy consumption ratio were equally likely), by 7% on average and by up to 15% for claude-opus-4-8 (latency 3.91, energy consumption 3.32), so a slow whole-program solution pays a smaller energy consumption penalty than its time penalty. This may be because the fastest solution of a CLBGComp cell draws more package power than the median of the other solutions in 49 of the 70 cells, for example when it uses both cores of the host.
For peak memory, the cost per attempted solution relates to the win probability on neither competition, neither in USD ( on LeetCodeComp and on CLBGComp) nor in tokens ( and ), with for all four.
The cost of a correct solution also depends on the language. For each LLM, we order the seven languages by its cost per correct solution, and a Friedman test [86] checks whether the LLMs agree on this order. They agree on both competitions, in USD ( on LeetCodeComp and 21.2 on CLBGComp) and in tokens ( and 20.5), with for all four. Python is the cheapest language on LeetCodeComp for 10 LLMs and PHP the most expensive for 12, in both USD and tokens. On CLBGComp, JavaScript is the cheapest for 7 LLMs and C++ the most expensive for 7, again in both units.
We also check whether the LLMs generate better or cheaper code in more popular languages, because popular languages may dominate the training data of the LLMs. We rank the languages by their use on GitHub in 2025 [20]. For each language, ાજ we compute the median over the LLMs of the ratio to the reference set in the language for each performance metric, the slowdown of the language against the fastest language on the same problems, the mean pass rate, and the median cost per correct solution in USD and in tokens. With seven languages, Spearman’s reaches significance at the 0.05 level only beyond . We find no evidence that popularity relates to any of these measures on either competition (, ). The strongest trends are that less popular languages cost more per correct solution on LeetCodeComp (, ) and have a lower pass rate there (, ). The cost trend is weaker on CLBGComp (, ).
References
- [1]M. Zeff, “Microsoft CEO says up to 30% of the company’s code was written by AI,” https://techcrunch.com/2025/04/29/microsoft-ceo-says-up-to-30-of-the-companys-code-was-written-by-ai/, 2025, techCrunch. Statement by Satya Nadella at Meta’s LlamaCon, April 29, 2025. Accessed September 2026.
- [2]M. Nazzaro, “Google CEO says more than 25 percent of company’s new code written by AI,” https://thehill.com/policy/technology/4962336-google-ceo-says-more-than-25-percent-of-companys-new-code-written-by-ai/, 2024, the Hill, October 30, 2024. Statement by Sundar Pichai on Alphabet’s Q3 2024 earnings call. Accessed September 2026.
- [3]E. Chakarian, “Google CEO Sundar Pichai says 75% of the company’s code is AI-generated,” https://www.fastcompany.com/91531519/google-ceo-says-75-of-the-companys-code-is-ai-generated, 2026, fast Company, April 23, 2026. Reporting on a blog post by Sundar Pichai. Accessed September 2026.
- [4]M. Chen, J. Tworek, H. Jun et al., “Evaluating large language models trained on code,” 2021, https://arxiv.org/abs/2107.03374.
- [5]J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023, pp. 21 558–21 572.arxiv.org/abs/2305.01210
- [6]N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “LiveCodeBench: Holistic and contamination free evaluation of large language models for code,” in International Conference on Learning Representations (ICLR), 2025, pp. 58 791–58 831.arxiv.org/abs/2403.07974
- [7]R. Rua and K. Ali, “Magnifying inefficiency: How LLMs amplify performance anti-patterns in mobile development,” in Journal Ahead Workshop (JAWS), 2026.
- [8]M. Singhal, T. Aggarwal, A. Awasthi, N. Natarajan, and A. Kanade, “NonfuEval: Funny how code LMs falter on requirements beyond functional correctness,” in First Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=h5umh6mzjarxiv.org/abs/2401.15963
- [9]M. Shetty, N. Jain, J. Liu, V. Kethanaboyina, K. Sen, and I. Stoica, “GSO: Challenging software optimization tasks for evaluating SWE-agents,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 38, 2025, pp. 167 761–167 804.
- [10]T. Cappendijk, P. de Reus, and A. Oprescu, “An exploration of prompting LLMs to generate energy-efficient code,” in 2025 IEEE/ACM 9th International Workshop on Green and Sustainable Software (GREENS). IEEE, 2025, pp. 31–38.DOI
- [11]D. Huang, Y. Qing, W. Shang, H. Cui, and J. M. Zhang, “EffiBench: Benchmarking the efficiency of automatically generated code,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 37, 2024, pp. 11 506–11 544.arxiv.org/abs/2402.02037
- [12]M. Du, A. T. Liu, B. Ji, Q. Liu, and S.-K. Ng, “Mercury: A code efficiency benchmark for code large language models,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 37, 2024, pp. 16 601–16 622.arxiv.org/abs/2402.07844
- [13]R. Qiu, W. Zeng, J. Ezick, C. Lott, and H. Tong, “How efficient is LLM-generated code? a rigorous & high-standard benchmark,” in International Conference on Learning Representations (ICLR), 2025, pp. 2233–2261.arxiv.org/abs/2406.06647
- [14]J. Liu, S. Xie, J. Wang, Y. Wei, Y. Ding, and L. Zhang, “Evaluating language models for efficient code generation,” in First Conference on Language Modeling (COLM), 2024. [Online]. Available: https://openreview.net/forum?id=IBCBMeAhmCarxiv.org/abs/2408.06450
- [15]Y. Qing, B. Zhu, M. Du, Z. Guo, T. Y. Zhuo, Q. Zhang, J. M. Zhang, H. Cui, S.-M. Yiu, D. Huang, S.-K. Ng, and L. A. Tuan, “EffiBench-x: A multi-language benchmark for measuring efficiency of LLM-generated code,” in Advances in Neural Information Processing Systems, vol. 38, 2025.
- [16]D. Ionescu, S. K. Jensen, and B. Thomsen, “Energy-bench: Holistic benchmark for correct and energy-efficient llm-generated code,” in 2025 3rd International Conference on Federated Learning Technologies and Applications (FLTA), 2025, pp. 571–579.DOI
- [17]M. A. Islam, D. V. Jonnala, R. Rekhi, P. Pokharel, S. Cilamkoti, A. Imran, T. Kasar, and B. O. Turkkan, “Evaluating the energy-efficiency of the code generated by LLMs,” in Proceedings of the International Conference on Foundation and Large Language Models, 2025, pp. 289–300.arxiv.org/abs/2505.20324
- [18]L. Solovyeva, S. Weidmann, and F. Castor, “AI-powered, but power-hungry? energy efficiency of LLM-generated code,” in IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge@ICSE), 2025, pp. 49–60.arxiv.org/abs/2502.02412
- [19]M. Balunović, J. De Koninck, I. Petrov, N. Jovanović, and M. Vechev, “MathArena: Evaluating LLMs on uncontaminated math competitions,” in Advances in Neural Information Processing Systems 38 (NeurIPS 2025) Datasets and Benchmarks Track, 2025, https://matharena.ai.arxiv.org/abs/2505.23281
- [20]GitHub, “Octoverse: A new developer joins GitHub every second as AI leads TypeScript to #1,” https://github.blog/news-insights/octoverse/octoverse-a-new-developer-joins-github-every-second-as-ai-leads-typescript-to-1/, 2025, accessed September 2026.
- [21]Benchmarks Game Team. (2026) The computer language benchmarks game. Accessed on 2026-09-18. [Online]. Available: https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html
- [22]LeetCode. (2026) LeetCode. Accessed on 2026-09-18. [Online]. Available: https://leetcode.com
- [23]R. Pereira, M. Couto, F. Ribeiro, R. Rua, J. Cunha, J. P. Fernandes, and J. Saraiva, “Energy efficiency across programming languages: How do energy, time, and memory relate?” in Proceedings of the 10th ACM SIGPLAN International Conference on Software Language Engineering, 2017, pp. 256–267.
- [24]R. Pereira, ——, “Ranking programming languages by energy efficiency,” Science of Computer Programming, vol. 205, p. 102609, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167642321000022
- [25]N. v. Kempen, H.-J. Kwon, D. T. Nguyen, and E. D. Berger, “It’s not easy being green: On the energy efficiency of programming languages,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Press, 2025, pp. 1553–1565. [Online]. Available: https://doi.org/10.1109/ASE636991.2025.00131
- [26]W. Oliveira, B. Moraes, F. Castor, and J. a. P. Fernandes, “Analyzing the resource usage overhead of mobile app development frameworks,” in Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’23. Association for Computing Machinery, 2023, pp. 152–161. [Online]. Available: https://doi.org/10.1145/3593434.3593487
- [27]Lim Yeri, “LeetCode_with_Solutions,” HuggingFace dataset, https://huggingface.co/datasets/LimYeri/LeetCode_with_Solutions.
- [28]GraphQL Foundation. GraphQL: A query language for apis. https://spec.graphql.org/.
- [29]Y. Xia, W. Shen, Y. Wang, J. K. Liu, H. Sun, S. Wu, J. Hu, and X. Xu, “LeetCodeDataset,” Hugging Face dataset, https://huggingface.co/datasets/newfacade/LeetCodeDataset, 2025.arxiv.org/abs/2504.14655
- [30]H. David, E. Gorbatov, U. R. Hanebutte, R. Khanna, and C. Le, “Rapl: Memory power estimation and capping,” in 2010 ACM/IEEE International Symposium on Low-Power Electronics and Design (ISLPED), Aug 2010, pp. 189–194.
- [31]CodeCarbon contributors, “CodeCarbon: Estimate and track carbon emissions from computing,” https://github.com/mlco2/codecarbon, 2024.
- [32]T. Sherwood, E. Perelman, G. Hamerly, and B. Calder, “Automatically characterizing large scale program behavior,” in Proceedings of the 10th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2002, pp. 45–57.
- [33]Linux kernel developers. perf: Linux profiling with performance counters. Linux perf_event subsystem. https://perfwiki.github.io/main/.
- [34]P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics, vol. 20, pp. 53–65, 1987.DOI
- [35]Anthropic, “Introducing Claude Opus 4.8,” https://www.anthropic.com/news/claude-opus-4-8, 2026, accessed October 2026.
- [36]OpenAI, “OpenAPI specification for the OpenAI API,” https://github.com/openai/openai-openapi, 2021, accessed September 2026.
- [37]Free Software Foundation. GNU time: Run programs and summarize system resource usage. https://www.gnu.org/software/time/.
- [38]Apple Inc. powermetrics: Power and thermal usage report tool. macOS command-line tool. https://www.unix.com/man-page/osx/1/powermetrics/.
- [39]OpenAI, “GPT-6 Astra model,” https://developers.openai.com/api/docs/models/gpt-6-astra, 2026, accessed October 2026.
- [40]——, “GPT-5.6 Sol model,” https://developers.openai.com/api/docs/models/gpt-5.6-sol, 2026, accessed October 2026.
- [41]——, “GPT-5.6 Luna model,” https://developers.openai.com/api/docs/models/gpt-5.6-luna, 2026, accessed October 2026.
- [42]——, “GPT-5.5 model,” https://developers.openai.com/api/docs/models/gpt-5.5, 2026, accessed October 2026.
- [43]——, “GPT-5.4 Mini model,” https://developers.openai.com/api/docs/models/gpt-5.4-mini, 2026, accessed October 2026.
- [44]Anthropic, “Introducing Claude Opus 5.5,” https://www.anthropic.com/claude-opus-5-5, 2026, accessed October 2026.
- [45]Anthropic, “Introducing Claude Sonnet 5.5,” https://www.anthropic.com/claude-sonnet-5-5, 2026, accessed October 2026.
- [46]Anthropic, “Introducing Claude Fable 5.1 and Claude Mythos 5.1,” https://www.anthropic.com/claude-fable-and-mythos-5-1, 2026, accessed October 2026.
- [47]Google DeepMind, “Gemini 3.8 Flash – model card,” https://deepmind.google/models/model-cards/gemini-3-8-flash/, 2026, accessed October 2026.
- [48]DeepSeek, “DeepSeek-V4.1-Flash: Smarter, faster, more efficient,” https://api-docs.deepseek.com/news/news260910/, 2026, accessed October 2026.
- [49]xAI, “Introducing Grok 4.6,” https://x.ai/news/grok-4-6, 2026, accessed October 2026.
- [50]Z.ai, “GLM-5.3-Flash/FlashX,” https://docs.z.ai/guides/vlm/glm-5.3-flash, 2026, accessed October 2026.
- [51]Google, “Developer’s guide to Gemini 3.8 Flash,” https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/guides/gemini-3-8-flash, 2026, accessed November 2026.
- [52]DeepSeek, “Thinking mode – DeepSeek API docs,” https://api-docs.deepseek.com/guides/thinking_mode, 2026, accessed October 2026.
- [53]OpenAI, “Using GPT-6,” https://developers.openai.com/api/docs/guides/latest-model, 2026, accessed October 2026.
- [54]Anthropic, “Thinking – Claude platform docs,” https://platform.claude.com/docs/en/build-with-claude/thinking, 2026, accessed October 2026.
- [55]Linux kernel developers. intel_pstate CPU performance scaling driver. The Linux Kernel documentation. https://docs.kernel.org/admin-guide/pm/intel_pstate.html.
- [56]K. N. Khan, M. Hirki, T. Niemi, J. K. Nurminen, and Z. Ou, “RAPL in action: Experiences in using RAPL for power measurements,” ACM Transactions on Modeling and Performance Evaluation of Computing Systems, vol. 3, no. 2, 2018.DOI
- [57]W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, and I. Stoica, “Chatbot Arena: An open platform for evaluating LLMs by human preference,” in Proceedings of the 41st International Conference on Machine Learning, ser. PMLR, vol. 235, 2024, pp. 8359–8388.arxiv.org/abs/2403.04132
- [58]R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. The method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952.DOI
- [59]D. R. Hunter, “MM algorithms for generalized Bradley–Terry models,” The Annals of Statistics, vol. 32, no. 1, pp. 384–406, 2004.
- [60]Standard Performance Evaluation Corporation, “SPEC CPU2026 benchmark suite,” 2026, https://www.spec.org/cpu2026/.
- [61]Doocs, “LeetCode solutions in any programming language,” https://github.com/doocs/leetcode, 2018, accessed October 2026.
- [62]S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979.DOI
- [63]M. D. Ernst, “Permutation methods: A basis for exact inference,” Statistical Science, vol. 19, no. 4, pp. 676–685, 2004.DOI
- [64]B. Efron, “Bootstrap methods: Another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, pp. 1–26, 1979.
- [65]M. G. Kendall, “A new measure of rank correlation,” Biometrika, vol. 30, no. 1–2, pp. 81–93, 1938.DOI
- [66]C. Spearman, “The proof and measurement of association between two things,” The American Journal of Psychology, vol. 15, no. 1, pp. 72–101, 1904. [Online]. Available: http://www.jstor.org/stable/1412159
- [67]M. Ehrgott, Multicriteria Optimization, 2nd ed. Berlin, Heidelberg: Springer, 2005.
- [68]E. Barrett, C. F. Bolz-Tereick, R. Killick, S. Mount, and L. Tratt, “Virtual machine warmup blows hot and cold,” Proceedings of the ACM on Programming Languages, vol. 1, no. OOPSLA, 2017.DOI
- [69]R. Apsan, V. Stoico, M. Albonico, R. Dhar, K. Vaidhyanathan, and I. Malavolta, “Generating energy-efficient code via large-language models - where are we now?” in Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, ser. ICSE ’26. New York, NY, USA: Association for Computing Machinery, 2026, pp. 426–439. [Online]. Available: https://doi.org/10.1145/3744916.3764522
- [70]D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in International Conference on Learning Representations (ICLR), 2021.
- [71]OpenAI, “Grade school math (GSM8K),” https://github.com/openai/grade-school-math, 2021.
- [72]W. Chi, V. Chen, A. N. Angelopoulos, W.-L. Chiang, A. Mittal, N. Jain, T. Zhang, I. Stoica, C. Donahue, and A. Talwalkar, “Copilot Arena: A platform for code LLM evaluation in the wild,” in Proceedings of the 42nd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, pp. 10 354–10 382. [Online]. Available: https://proceedings.mlr.press/v267/chi25a.htmlarxiv.org/abs/2502.09328
- [73]T. Coignion, C. Quinton, and R. Rouvoy, “A performance study of LLM-generated code on LeetCode,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 2024, pp. 79–89.DOI
- [74]V.-A. Cursaru, L. Duits, J. Milligan, D. Ural, B. Rodriguez Sanchez, V. Stoico, and I. Malavolta, “A controlled experiment on the energy efficiency of the source code generated by Code Llama,” in Quality of Information and Communications Technology (QUATIC 2024), ser. Communications in Computer and Information Science, A. Bertolino, J. Pascoal Faria, P. Lago, and L. Semini, Eds., vol. 2178. Cham: Springer, 2024, pp. 161–176.arxiv.org/abs/2405.03616
- [75]T. Vartziotis, I. Dellatolas, G. Dasoulas, M. Schmidt, F. Schneider, T. Hoffmann, S. Kotsopoulos, and M. Keckeisen, “Learn to code sustainably: An empirical study on green code generation,” in Proceedings of the 1st International Workshop on Large Language Models for Code (LLM4Code). ACM, 2024, pp. 30–37.DOI
- [76]C. Niu, T. Zhang, C. Li, B. Luo, and V. Ng, “On evaluating the efficiency of source code generated by LLMs,” in Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering (FORGE). ACM, 2024, pp. 103–107.arxiv.org/abs/2404.06041
- [77]Y. Peng, J. Wan, Y. Li, and X. Ren, “COFFE: A code efficiency benchmark for code generation,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 242–265, 2025.arxiv.org/abs/2502.02827
- [78]A. Shypula, A. Madaan, Y. Zeng, U. Alon, J. Gardner, Y. Yang, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, “Learning performance-improving code edits,” in The Twelfth International Conference on Learning Representations (ICLR), 2024, pp. 47 277–47 301. [Online]. Available: https://openreview.net/forum?id=ix7rLVHXyYarxiv.org/abs/2302.07867
- [79]S. Waghjale, V. Veerendranath, Z. Wang, and D. Fried, “ECCO: Can we improve model-generated code efficiency without sacrificing functional correctness?” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, 2024, pp. 15 362–15 376.
- [80]X. He, Q. Liu, M. Du, L. Yan, Z. Fan, Y. Huang, Y. Zheng, Z. Yuan, and Z. Ma, “SWE-Perf: Can language models optimize code performance on real-world repositories?” in Proceedings of the 43rd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 306. PMLR, 2026, pp. 41 651–41 666.arxiv.org/abs/2507.12415
- [81]J. J. Ma, M. Hashemi, A. Yazdanbakhsh, K. Swersky, O. Press, E. Li, V. J. Reddi, and P. Ranganathan, “SWEfficiency: Can language models optimize real-world repositories on real workloads?” in Proceedings of the 43rd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 306. PMLR, 2026, pp. 84 389–84 427.arxiv.org/abs/2511.06090
- [82]A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré, and A. Mirhoseini, “KernelBench: Can LLMs write efficient GPU kernels?” in Proceedings of the 42nd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, pp. 47 356–47 415. [Online]. Available: https://proceedings.mlr.press/v267/ouyang25a.htmlarxiv.org/abs/2502.10517
- [83]M. Du, A. T. Luu, X. Liu, Y. Qing, D. Huang, X. He, Q. Liu, Z. Ma, and S.-K. Ng, “Afterburner: Reinforcement learning facilitates self-improving code efficiency optimization,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 38, 2025, pp. 9944–9976.arxiv.org/abs/2505.23387
- [84]D. Huang, J. Dai, H. Weng, P. Wu, Y. Qing, H. Cui, Z. Guo, and J. M. Zhang, “EffiLearner: Enhancing efficiency of generated code via self-optimization,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 37. Curran Associates, Inc., 2024, pp. 84 482–84 522.arxiv.org/abs/2405.15189
- [85]Z. Chen, S. Fang, and M. Monperrus, “Supersonic: Learning to generate source code optimizations in C/C++,” IEEE Transactions on Software Engineering, vol. 50, no. 11, pp. 2849–2864, 2024.DOI
- [86]M. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,” Journal of the American Statistical Association, vol. 32, no. 200, pp. 675–701, 1937.DOI
