betaFeedback

WorksComputer Science

Beyond Accuracy: How Quantization Affects Coding Agents

Abstract

Agents are increasingly used for coding tasks, but the inference cost of large language models (LLMs) limits where they can run. Quantization reduces model size, but its effects on coding agents that execute and revise programs remain underexplored. We compare BF16 with 4-bit AWQ and GPTQ for Qwen2.5-Coder-Instruct (7B, 14B, and 32B) and Devstral-Small-2507 (24B) on OJBench coding problems across three seeds. We detect no statistically significant accuracy loss for any of the coding agents. However, final accuracy can conceal substantial behavioral changes: AWQ reduces Qwen-7B one-shot solves from 17 to 0 while increasing loop-based solves from 10 to 25. All Qwen-32B successes involve tool use, whereas Devstral rarely makes a valid executor call. Failure profiles also differ: Qwen-32B produces far more compilation errors than the smaller Qwen models, rising slightly under quantization, while Devstral's failures shift significantly from runtime errors toward wrong answers and timeouts. These findings indicate that quantization robustness in coding agents should be assessed through solution trajectories, tool use, and failure modes, in addition to final accuracy.

Introduction

Large Language Models (LLMs) [7, 16] are increasingly used in agentic coding systems that go beyond single-pass text generation. These agents can invoke execution tools, inspect program outputs, and revise their solutions over multiple steps [5, 4, 9, 2]. This interaction changes how coding capability should be evaluated: a model that produces an incorrect program initially may still reach a correct solution after execution feedback. Recent benchmarks such as OJBench [18] provide challenging settings for studying this behavior. OJBench contains competition-level programming problems and places greater emphasis on reasoning and code correctness than typical interview-style benchmarks.

Efficient deployment of these agents remains challenging because repeated LLM inference and tool interaction increase computational and memory costs. Quantization is a widely used approach for reducing these costs [3, 19, 11]. In particular, weight-only methods such as AWQ and GPTQ can substantially reduce model storage while retaining high-precision activations. However, reasoning performance can be sensitive to low-bit quantization, especially on difficult tasks where small perturbations can affect multi-step generation [12]. Most existing studies evaluate this effect using single-pass model outputs. Coding agents introduce a different setting because quantization can influence not only the generated program, but also tool invocation, iterative refinement, and the final failure mode.

This distinction raises an important question: does similar final accuracy imply that a quantized coding agent behaves similarly to its full-precision counterpart? Our results suggest that it does not. Quantization can preserve final accuracy while changing solve trajectories, tool-use patterns, and terminal failure modes. These effects are hidden when coding agents are evaluated only by their final score. In this work, we present, to the best of our knowledge, the first empirical study of 4-bit weight-only quantization in execution-feedback coding agents. We evaluate BF16, AWQ, and GPTQ versions of Qwen2.5-Coder-Instruct [6] at 7B, 14B, and 32B scales, together with Devstral-Small-2507 [13]. All models use vLLM with an OpenAI-compatible tool schema and a Python execution tool and are evaluated over three random seeds on OJBench. We investigate three research questions:

RQ1: How robust is end-to-end coding-agent performance to 4-bit quantization?

RQ2: Does end-to-end accuracy conceal changes in coding-agent solution trajectories?

RQ3: Does quantization change how coding agents fail, not just how often?

Related Work

Recent work on coding agents extends single-pass code generation by incorporating program execution and iterative revision. Studies on self-debugging show that LLMs can use execution feedback to identify and correct errors in generated programs [2]. Existing evaluations cover real-world data-science tasks through DSBC [Kadiyala et al., 2025], automated program repair [14], interactive application environments such as AppWorld [17], multilingual repository-level tasks through SWE-PolyBench [15], and competition-level programming through OJBench [18]. AWQ [10] and GPTQ [3] are widely used methods for 4-bit weight-only quantization. Prior single-pass studies indicate that quantization generally preserves reasoning performance, although smaller models and more difficult tasks experience greater degradation [12]. However, these studies do not examine how quantization affects coding agents that can execute and revise their outputs. We address this gap by asking whether the execution loop absorbs quantization-induced errors, and whether the size–robustness ordering observed in single-pass settings still holds when the agent can run its code.

Experimental Setup

This section describes the setup used to evaluate BF16 and AWQ/GPTQ 4-bit quantized models on competitive coding tasks within an execution-based agentic workflow.

Dataset: OJBench

We evaluate on OJBench [18], a benchmark of 232 competition-level problems from China’s National Olympiad in Informatics (NOI) and the International Collegiate Programming Contest (ICPC). OJBench supports solutions in both Python and C++; we use Python only, so all solutions are written, executed, and judged in Python. We use all 115 Easy and Medium problems because all model–precision configurations scored 0% on Hard. Evaluation across three seeds yields 345 problem–seed observations per configuration.

Evaluation Setup

Models and Quantization. We evaluate Qwen2.5-Coder-Instruct [6] at 7B, 14B, and 32B, spanning scales at which size may interact with quantization, plus the agent-oriented 24B Devstral-Small-2507 [13] as an out-of-family comparison. Each model is run at BF16 and as W4A16 variants produced with AWQ [10] and GPTQ [3] via llmcompressor [1] (256 calibration samples, max sequence length 1,024).

Agent Framework. Each model is served with vLLM in a multi-turn execution-feedback loop using an OpenAI-compatible tool schema, with tool calls parsed from the generated text. The agent has one tool, python_executor(code, stdin). It writes code, tests it on the problem’s examples, reads the feedback, and revises for up to five tool calls per problem. All runs use nucleus sampling (temperature 0.6, top-p 0.9), 32,768-token context, and three seeds. Full prompt and loop in Appendices A.3 and A.4.

Evaluation Metrics. Accuracy is the proportion of final programs passing all official OJBench test cases. Each failed attempt is assigned one terminal category: Invalid Return (IR; runtime exception), Wrong Answer (WA), Time Limit Exceeded (TLE), Compilation Error (CE), or Memory Limit Exceeded (MLE). We also record tool calls per problem, distinguishing one-shot solves (no tool calls) from loop-based solves (at least one execution round). Because every precision is evaluated on the same problems and seeds, outcomes are paired; we apply an exact two-sided McNemar test to the 345 paired observations, with bb and cc the discordant pairs solved by only the first or only the second precision, and p<0.05p < 0.05 as the threshold. For the behavioral measures, we use Fisher’s exact test for solve paths and tool-use rates, and a chi-square test for failure distributions. Because each analysis involves eight BF16-versus-quantized comparisons, we also report Holm-adjusted pp-values. Because the three seeds of a problem are correlated, we additionally treat each problem as one unit: we report 95% confidence intervals from a cluster bootstrap over the 115 problems (20,000 resamples, keeping all three seeds of a problem together) and pp-values from a permutation test that swaps precision labels within each problem.

Empirical Findings

We investigate how weight-only quantization affects Python coding performance on OJBench for execution-feedback agents across BF16, AWQ, and GPTQ 4-bit precisions. Table 1 reports accuracy, Figure 1 the failure-mode distribution, and Appendix A.2 the paired tests and tool-use statistics.

ModelPrecisionEasy (%)Medium (%)Overall (%)
Qwen-2.5-Coder-7BBF1621.29 ±1.6\pm1.61.69 ±0.7\pm0.77.83 ±0.9\pm0.9
Qwen-2.5-Coder-7BAWQ20.37 ±1.6\pm1.61.27 ±0.0\pm0.07.25 ±0.5\pm0.5
Qwen-2.5-Coder-7BGPTQ18.52 ±3.2\pm3.21.27 ±0.0\pm0.06.67 ±1.0\pm1.0
Qwen-2.5-Coder-14BBF1630.56 ±2.8\pm2.82.53 ±1.3\pm1.311.30 ±0.0\pm0.0
Qwen-2.5-Coder-14BAWQ26.85 ±1.6\pm1.61.27 ±0.0\pm0.09.28 ±0.5\pm0.5
Qwen-2.5-Coder-14BGPTQ26.85 ±8.9\pm8.92.11 ±0.7\pm0.79.86 ±2.7\pm2.7
Qwen-2.5-Coder-32BBF1632.41 ±1.6\pm1.65.48 ±3.2\pm3.213.91 ±2.3\pm2.3
Qwen-2.5-Coder-32BAWQ25.93 ±5.8\pm5.83.80 ±0.0\pm0.010.72 ±1.8\pm1.8
Qwen-2.5-Coder-32BGPTQ30.56 ±0.0\pm0.02.11 ±0.7\pm0.711.01 ±0.5\pm0.5
Devstral-Small-2507BF1611.11 ±2.8\pm2.81.69 ±0.7\pm0.74.64 ±1.3\pm1.3
Devstral-Small-2507AWQ15.74 ±8.5\pm8.51.69 ±0.7\pm0.76.09 ±2.6\pm2.6
Devstral-Small-2507GPTQ13.89 ±2.8\pm2.81.69 ±0.7\pm0.75.51 ±1.3\pm1.3

Table 1. Performance of Qwen-2.5-Coder and Devstral-Small-2507 agents (vLLM, OpenAI-compatible tool schema) on OJBench, reported as mean±\pmstd over 3 seeds. Darker shading marks the best overall accuracy per model family, lighter the second best. Overall accuracy is over the Easy and Medium splits (115 problems); the Hard split is excluded because all models scored 0%.

Distribution of failure categories across BF16, AWQ, and GPTQ for each agent

Figure 1. Distribution of failure categories across BF16, AWQ, and GPTQ for each agent. Categories: IR (Invalid Return), WA (Wrong Answer), TLE (Time Limit Exceeded), CE (Compilation Error), MLE (Memory Limit Exceeded). Counts are per-seed averages over the Easy+Medium splits (115 problems, 3 seeds).

RQ1: How robust is end-to-end coding-agent performance to 4-bit quantization?

We evaluate whether replacing BF16 weights with W4A16 AWQ or GPTQ changes final coding accuracy in execution-feedback agents. We compare accuracy across Qwen-2.5-Coder scales and Devstral-Small-2507, together with paired statistical tests in Table 3. This establishes whether quantization-induced perturbations remain visible after the complete agentic execution process and whether the effect is consistent across quantizers and model configurations. Because we run eight BF16-versus-quantized comparisons, we apply a Holm correction, and because the three seeds of a problem are correlated, we also repeat the analysis with each problem as one unit. Under both analyses, no comparison shows a statistically significant accuracy loss. Qwen-32B shows the largest drop, about 3 points under both quantizers, but it does not reach significance after correction (pHolm=0.21p_{\mathrm{Holm}} = 0.21 for AWQ) or at the problem level (p=0.051p = 0.051).

ModelSchemebbccpppHolmp_{\mathrm{Holm}}Δ\Delta95% CIpprobp_{\mathrm{prob}}
Qwen-2.5-Coder-7BAWQ860.791.000.58[−1.74, 3.19]0.82
GPTQ840.391.001.16[−0.87, 3.19]0.42
Qwen-2.5-Coder-14BAWQ920.0650.462.03[0.00, 4.06]0.094
GPTQ940.271.001.45[−0.58, 3.48]0.27
Qwen-2.5-Coder-32BAWQ1650.0270.213.19[0.58, 6.09]0.051
GPTQ1880.0760.462.90[−0.29, 6.38]0.13
Devstral-Small-2507AWQ7120.361.00−1.45[−3.48, 0.58]0.27
GPTQ8110.651.00−0.87[−3.48, 1.45]0.65

Table 3. Exact two-sided McNemar tests, BF16 versus quantized, over paired per-problem outcomes (same problem, same seed; 345 pairs per comparison). bb = problems solved only by BF16; cc = problems solved only by the quantized model. pHolmp_{\mathrm{Holm}} applies a Holm correction across the eight comparisons. The 32B/AWQ comparison is significant before correction (pp = 0.027, bold) but not after it; no comparison is significant after correction. The last three columns treat each problem as one unit: Δ\Delta is BF16 minus quantized accuracy (points), the 95% CI comes from a cluster bootstrap over the 115 problems, and pprobp_{\mathrm{prob}} from a within-problem permutation test. The problem-level analysis gives the same conclusion: no comparison is significant at α\alpha = 0.05, although the 32B/AWQ result is borderline (its CI just excludes zero, pp = 0.051).

Finding: Quantization has limited effect on final accuracy for all agents. Qwen-2.5-Coder-32B shows the largest drop, but it does not reach statistical significance.

RQ2: Does end-to-end accuracy conceal changes in coding-agent solution trajectories?

We analyze successful trajectories using one-shot solves, execution-loop solves, tool-use rate, and mean executor calls (Table 4 and Table 5). Across precisions, Qwen-7B shows the clearest behavioral shift: one-shot solves fall from 17 under BF16 to 0 under AWQ and 3 under GPTQ (AWQ: 17 → 0; GPTQ: 17 → 3), while loop-based solves rise from 10 under BF16 to 25 under AWQ and 20 under GPTQ (AWQ: 10 → 25; GPTQ: 10 → 20). This shift remains significant when each problem is treated as one unit, and it also holds on problems solved by both precisions. Devstral moves in the opposite direction, with valid tool use falling from 8.7% under BF16 to 1.2% under GPTQ (GPTQ: 8.7% → 1.2%), where all 19 successful solves are one-shot. Malformed tool calls are rare for the Qwen models, but most of Devstral’s tool calls do not follow the tool schema, which keeps its recorded tool use low; quantization still reduces how often it tries the tool (AWQ: 42.0% → 27.0%; GPTQ: 42.0% → 21.4%). The Qwen models also differ from each other at BF16: Qwen-32B uses tools on 71.3% of problems and produces no one-shot solves, whereas Qwen-14B uses tools on 31.6% of problems with a near-balanced split of 21 one-shot and 18 loop-based solves. These results show both model-specific solve strategies and different levels of behavioral stability under quantization.

ModelPrecisionUsed tool (%)Tried tool (%)Malformed (%)Mean callsOne-shot solvesLoop solves
Qwen-2.5-Coder-7BBF1666.770.13.82.261710
Qwen-2.5-Coder-7BAWQ73.982.9†5.92.29025
Qwen-2.5-Coder-7BGPTQ69.071.93.41.90320
Qwen-2.5-Coder-14BBF1631.631.60.01.212118
Qwen-2.5-Coder-14BAWQ26.726.70.01.011715
Qwen-2.5-Coder-14BGPTQ34.835.90.81.551717
Qwen-2.5-Coder-32BBF1671.371.60.13.07048
Qwen-2.5-Coder-32BAWQ63.864.30.42.69037
Qwen-2.5-Coder-32BGPTQ67.069.60.93.15038
Devstral-Small-2507BF168.742.076.90.11133
Devstral-Small-2507AWQ8.427.0†69.30.09174
Devstral-Small-2507GPTQ1.221.4†96.50.01190

Table 4. Tool use and solve path per agent, totals over 3 seeds (345 attempts). “Used tool” is the share of problems with at least one valid executor call. “Tried tool” also counts attempts in which the model issued a tool call that did not follow the provided tool schema (e.g., broken JSON or the tool written as a Python function inside its code). “Malformed” is the share of all tool calls that did not follow the schema (mean over seeds). † marks tried-tool rates that differ significantly from BF16 after Holm correction. “One-shot” solves used zero tool calls; “loop” solves followed at least one execute-and-check round.

ModelSchemeSolve path ppSolve path pHolmp_{\mathrm{Holm}}Tool use ppTool use pHolmp_{\mathrm{Holm}}
Qwen-2.5-Coder-7BAWQ<0.001<0.001<0.001<0.0010.0450.29
Qwen-2.5-Coder-7BGPTQ<0.001<0.0010.0030.571.00
Qwen-2.5-Coder-14BAWQ1.001.000.180.90
Qwen-2.5-Coder-14BGPTQ0.821.000.421.00
Qwen-2.5-Coder-32BAWQ1.001.000.0420.29
Qwen-2.5-Coder-32BGPTQ1.001.000.250.99
Devstral-Small-2507AWQ1.001.001.001.00
Devstral-Small-2507GPTQ0.0860.51<0.001<0.001<0.001<0.001

Table 5. Fisher’s exact tests on the behavioral measures in Table 4, BF16 versus quantized. Solve path tests whether the one-shot/loop split among solved problems differs from BF16; tool use tests whether the share of problems with at least one executor call differs from BF16. pHolmp_{\mathrm{Holm}} applies a Holm correction across the eight comparisons within each measure. Significant results after correction are in bold. The tests treat runs as independent, so pp-values are approximate. At the problem level, the Qwen-7B solve-path shift is significant under AWQ (p=0.005p = 0.005) and GPTQ (p=0.022p = 0.022). On problems solved by both precisions, every BF16 one-shot solve becomes a loop solve under AWQ (8 of 8 problems, p=0.008p = 0.008), and 7 of 8 under GPTQ (p=0.070p = 0.070), which does not reach significance on its own.

Finding: Models exhibit distinct tool-use regimes, but quantization does not preserve them uniformly. Qwen-32B remains loop-dependent, Qwen-14B stays relatively balanced, Qwen-7B shifts toward loop-based solving, and Devstral becomes almost entirely one-shot under GPTQ.

RQ3: Does quantization change how coding agents fail, not just how often?

We compare terminal failure categories (IR /WA /TLE /CE /MLE) across precision settings, and test whether each failure distribution differs from BF16 using a chi-square test with Holm correction (Table 2). For Qwen-7B, AWQ shifts failures slightly away from Wrong Answer and toward Invalid Return (WA: 65.0 → 59.7; IR: 21.7 → 29.0 per seed). This suggests that quantization can change the type of terminal failure even when final accuracy changes little. The pattern differs for larger models. Qwen-32B exhibits substantially more Compilation Errors than the smaller Qwen models, with CE rising slightly under quantization. Devstral follows another pattern, with fewer Invalid Returns but more Wrong Answers and Time Limit Exceeded cases after quantization. This is the clearest change, and it is statistically significant under both quantizers.

ModelPrecisionIRWATLECEMLEpppHolmp_{\mathrm{Holm}}
Qwen-2.5-Coder-7BBF1621.765.017.01.30.7––
AWQ29.059.715.02.30.70.170.84
GPTQ23.366.015.02.01.00.811.00
Qwen-2.5-Coder-14BBF1615.366.318.02.00.3––
AWQ13.069.017.04.31.00.341.00
GPTQ21.061.318.72.00.70.351.00
Qwen-2.5-Coder-32BBF166.763.715.013.30.3––
AWQ7.766.710.717.70.00.240.97
GPTQ5.758.716.720.70.70.130.75
Devstral-Small-2507BF1653.344.77.04.70.0––
AWQ37.351.010.09.00.70.0020.011
GPTQ39.351.714.33.30.00.0010.009

Table 2. Distribution of Failure Types in Agent Submissions. IR: Invalid Return, WA: Wrong Answer, TLE: Time Limit Exceeded, CE: Compilation Error, MLE: Memory Limit Exceeded. Counts are per-seed averages over the Easy+Medium splits (115 problems, 3 seeds). pp is a chi-square test of whether the failure distribution differs from BF16, computed on totals over the three seeds; MLE is excluded from the test because its counts are very small. pHolmp_{\mathrm{Holm}} applies a Holm correction across the eight comparisons.

Finding: Quantization can reshape failure profiles differently across models rather than producing one universal failure pattern.

Conclusion

We evaluated 4-bit weight-only quantization in execution-feedback coding agents on OJBench, comparing BF16, AWQ, and GPTQ across Qwen2.5-Coder (7B, 14B, 32B) and Devstral-Small-2507. Quantization reduces checkpoint storage by about 3.4×3.4 \times and causes no statistically significant accuracy loss for any agent after correcting for multiple comparisons, although Qwen-32B shows a consistent nominal drop. More importantly, similar final accuracy can hide large behavioral shifts: Qwen-7B moves strongly from one-shot to loop-based solving after quantization, while Qwen-32B remains fully loop-dependent. Quantization also changes terminal failure profiles differently across models. These results show that quantized coding agents should be evaluated through accuracy, solve trajectories, tool use, and failure modes rather than final accuracy alone.

Limitations and future work. Our study focuses on one competition-level benchmark, one execution-feedback scaffold, and W4A16 quantization, with Python as the solution language. Because OJBench is challenging, absolute solve counts are modest, so we emphasize effects that hold at the problem level. Extending this analysis to other benchmarks, languages, and bit-widths, and to end-to-end efficiency measures such as latency and memory, is a natural next step. Within these limits, 4-bit quantization appears practical for execution-feedback coding agents, but its reliability should be evaluated within the complete agentic workflow.

Acknowledgments

The authors used Claude Opus 5.5 for language and formatting assistance. All content was reviewed by the authors, who remain responsible for the final manuscript.

Appendix

Error Analysis

Statistical Tests and Solve Path

Overall agent accuracy on OJBench for Qwen-7B, Qwen-14B, Qwen-32B, and Devstral

Figure 2. Overall agent accuracy on OJBench (Easy+Medium, 115 problems), mean over 3 seeds with one standard deviation. The 32B agent under AWQ is the only comparison that differs from BF16 before correction (paired McNemar, *); after Holm correction across the eight comparisons, no gap is statistically significant.

Coding Agent Execution Prompt

The following system prompt was used to instruct the coding agent during problem solving. It guides the agent through a structured write–test–fix workflow with code execution for verification.

You are a competitive programming assistant. Solve problems
step-by-step:
1. Read the problem carefully
2. Write Python code to solve it
3. Create test input from the problem’s examples
4. Call python_executor tool to test your code with the test
input
5. Check if the output matches expected output
6. If wrong, fix the code and call python_executor again
7. Once correct, return your final solution in a Python code
block
Input reading patterns:
- int(input()) for single number
- map(int, input().split()) for multiple numbers on one
line
Always test your code before giving the final answer!

The python_executor(code, stdin) tool is not described in the prompt text itself. It is supplied to the model through the chat template’s OpenAI-compatible tool schema, and tool calls are parsed from the generated text. The tool description reads: “Execute Python code with optional stdin input. Use this to test your solution before giving the final answer.” The 30 s execution timeout and the five-call budget are enforced by the harness and are not disclosed to the model in advance. When the budget is reached, the model receives one additional user message: “You have reached the maximum number of tool calls. Please provide your final solution in a Python code block now.”

Algorithm

Each turn consists of a single model generation. If the generation contains no tool call, it is taken as the final answer. Otherwise every tool call in the generation is executed and its output, truncated to 2,000 characters, is appended to the conversation. The tool-call budget KK is checked once per turn rather than once per call, so a turn containing several calls can carry the total past KK; this occurred in roughly 10% of problem attempts. When the budget is reached, the model is prompted once more to emit its final solution. If the accumulated conversation exceeds the 32,768-token context limit, the oldest assistant and tool messages are dropped before generation.

Algorithm 1 Execution-Guided Coding Agent with Tool Feedback
Require: Prompt dataset $D$, model $M$, python executor $E$, tool-call budget $K = 5$
Ensure: Final solution and tool-call count for each problem
1: for each prompt $x \in D$ do
2:     messages $\gets$ [system prompt, $x$]
3:     tool_calls $\gets 0$
4:     final_response $\gets \emptyset$
5:     for turn $= 1, \ldots, K + 1$ do
6:         Trim oldest assistant and tool messages if context budget is exceeded
7:         Generate response $r$ from model $M$ on messages
8:         Parse tool calls $T$ from the text of $r$
9:         if $T$ is empty then
10:             final_response $\gets r$
11:             Break loop
12:         end if
13:         for each call $t \in T$ do
14:             tool_calls $\gets$ tool_calls $+ 1$
15:             Execute $t$ using executor $E$ and truncate output to 2,000 characters
16:             Append $r$ and execution result to messages
17:         end for
18:         if tool_calls $\ge K$ then
19:             Append user message requesting the final solution
20:             Generate final_response from model $M$ on messages
21:             Break loop
22:         end if
23:     end for
24:     if final_response is empty then
25:         final_response $\gets$ last $r$, or error message if no generation succeeded
26:     end if
27:     Save result including metadata and tool_calls
28: end for

References

  1. [1]Red Hat AI and vLLM Project. LLM Compressor, 8 2024. URL https://github.com/vllm-project/llm-compressor.
  2. [2]Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In International Conference on Learning Representations (ICLR), 2024.
  3. [3]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. URL https://arxiv.org/abs/2210.17323.
  4. [4]Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. arXiv preprint arXiv:2402.17453, 2024.arxiv.org/abs/2402.17453
  5. [5]Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, et al. Data interpreter: An llm agent for data science. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19796–19821, 2025.arxiv.org/abs/2402.18679
  6. [6]Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shangchuan Quan, Yunlong Feng, Xingzhan Ren, Xuanchen Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. URL https://arxiv.org/abs/2409.12186.
  7. [7]Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.arxiv.org/abs/2412.16720
  8. [9]Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR, 2023.arxiv.org/abs/2211.11501
  9. [10]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024. URL https://arxiv.org/abs/2306.00978.
  10. [11]Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhou Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.01241, 2024.arxiv.org/abs/2403.01241
  11. [12]Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiez­heng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models, 2025. URL https://arxiv.org/abs/2504.04823.
  12. [13]Mistral AI. Upgrading agentic coding capabilities with the new devstral models. https://mistral.ai/news/devstral-2507/, 2025. Devstral Small 1.1 (devstral-small-2507).
  13. [14]Meghana Puvvadi, Sai Kumar Arava, Adarsh Santoria, Sesha Sai Prasanna Chennupati, and Harsha Vardhan Puvvadi. Coding agents: A comprehensive survey of automated bug fixing systems and benchmarks. In 2025 IEEE 14th International Conference on Communication Systems and Network Technologies (CSNT), pages 680–686, 2025. doi: 10.1109/CSNT64827.2025.10968728.DOI
  14. [15]Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents, 2025. URL https://arxiv.org/abs/2504.08703.
  15. [16]Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.
  16. [17]Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents, 2024. URL https://arxiv.org/abs/2407.18901.
  17. [18]Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, Tianyu Liu, and Weiran Xu. Objbench: A competition level code benchmark for large language models, 2025. URL https://arxiv.org/abs/2506.16395.
  18. [19]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pages 38087–38099. PMLR, 2023.arxiv.org/abs/2211.10438

Paper details

vlnc:W_fnmn5xgw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 24

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Opus 5.5

View statement
Looking for related papers…
Contents