betaFeedback

Reading v1View latest version

WorksComputer Science

CompressionBench: How can your coding agents compress LLMs?

Abstract

Recursive self-improvement requires AI systems to improve the models and infrastructure used by subsequent systems. LLM compression offers a concrete efficiency objective: retain a pretrained model's behavior within a smaller model storage budget. We study whether coding agents can carry out this step autonomously. We formulate compression as a terminal task in which an agent receives a source checkpoint, storage limit, inference environment, and time budget, then constructs compressed weights and executable inference code. The agent controls its algorithm and experiments; perplexity feedback guides search, and independent verification checks the final model. Across six agents and six model–size settings, 29 of 36 submissions pass verification. Selecting the lowest-perplexity agent design per task yields an average 3.4% perplexity increase over BF16 and 77.33% downstream accuracy, compared with 79.56% for BF16. Recorded experiments reveal how agents achieve these outcomes and where improvement breaks down. Agents lower PPL in 21 of 24 revisions at nearly fixed storage; complete-model feedback reverses 17 of 79 prior local rankings; and generation and inference tests expose failures that likelihood alone misses. The results identify representation design, experimental judgment, and capability validation as central to agent-driven model compression, a measurable component of efficiency-oriented self-improvement.

Equal contribution.

Code github.com/Infini-AI-Lab/Compression-Bench

Blog infini-ai-lab.github.io/Compression-Bench

Introduction

Large language model (LLM) inference is increasingly bounded by memory traffic rather than arithmetic: in autoregressive decoding, every generated token reads the model weights from GPU memory, and an off-chip memory access costs far more energy than the arithmetic it feeds Horowitz, [16]. Compression that preserves a model’s behavior within a smaller storage budget therefore reduces both the memory a deployment occupies and the energy spent on every token it generates, making models cheaper to serve and runnable on smaller hardware [12] (Lin et al. 2024; Lin et al. 2025). Constructing such a model remains substantial experimental and systems work, the kind of work that coding agents increasingly undertake (Novikov et al., 2025; Zhang et al., 2026). An agent that compresses models would improve the efficiency of the AI systems that follow it, a concrete step toward recursive self-improvement. We therefore ask: can coding agents autonomously construct compressed LLMs that remain useful within a given storage budget?

Existing compression methods provide a rich set of building blocks, but no single method yields a useful model under an arbitrary storage limit and hardware target. To construct one autonomously, an agent needs three abilities:

  • Coordinating coupled numerical decisions. A storage budget ties together representation, precision allocation, and parameter retention: bits spent on one component are unavailable to the rest, and pruning releases bytes but removes capacity. Even at the same nominal precision, clipping, rounding, calibration, and metadata change the quality of a representation [12] (Shao et al. 2024). The agent must judge feasible combinations rather than optimize each choice independently.

  • Interpreting incomplete feedback. Weight reconstruction error, layer-output error, complete-model likelihood, and generated-answer accuracy each answer a different question. A change that improves a local approximation can damage the complete model, and a small likelihood gain can conceal a generation regression. The agent must decide when inexpensive local evidence suffices and when a complete-model experiment is worth its cost.

  • Co-designing efficient execution. Compression saves memory and energy only when unpacking, reconstruction, and indexing do not erase the reduction in memory traffic, and persistent dequantized copies can erase the resident-storage savings of a small checkpoint. The agent must implement an inference path suited to the target hardware, as in quantized inference systems (Lin et al., 2024, 2025), and verify that it runs correctly.

To evaluate these abilities without prescribing how agents compress, we formulate LLM compression as a terminal task (Figure 2), following the outcome-based evaluation of Terminal-Bench and Harbor [13] (Merrill et al. 2026). A coding agent reads papers, writes code, constructs checkpoints, and runs experiments within a fixed time budget. Three designs support this evaluation. 1. Task: a complete model under a storage limit. Given source weights, a storage limit, a GPU, and a pinned runtime, the agent may combine quantization, mixed precision, pruning, low-rank representations, and distillation, and must implement the inference operations its representation requires (§ 2.1). 2. Feedback: complete-model likelihood. A supplied evaluator returns the source-to-compressed perplexity (PPL) ratio on continuations sampled from the source model; the expected excess cross-entropy equals the next-token KL divergence from the source (§ 2.3). Evaluations share the agent’s time budget, so the agent decides when local evidence suffices and when a complete-model measurement is worth its cost.

Figure 2. Compression through terminal interaction. A task fixes the source checkpoint, storage allowance CC, GPU and runtime, and search budget TT. The agent constructs and tests compressed models, using reward feedback to revise weights and inference code. At the deadline, a separate evaluator checks the frozen artifact’s likelihood, generated answers, storage, and execution. Decode speed (TPS, batch size one) is reported separately from reward.

3. Validation: the executable artifact. Independent verification runs the submitted weights through the submitted inference code on held-out contexts. Both the saved package and the resident GPU weights must fit the limit; GSM8K, HumanEval, and TriviaQA accuracy must retain 80% of BF16; and review checks integrity. Decode speed is reported separately (§ 2.4). Only verified models receive a score.

In § 3, we evaluate the models that six coding agents deliver for three source models with two storage limits each. Of the 36 resulting models, 29 pass verification, including all 18 produced under the looser limits. Selecting the lowest-PPL agent design per task yields an average 3.4% PPL increase over BF16 and 77.33% downstream accuracy, compared with 79.56% for BF16 (§ 3.2). In § 4, we analyze the agents’ recorded searches to explain how they improve compression and where improvement breaks down, yielding three findings. Representation design creates room at fixed storage. In 21 of 24 revisions that change package size by at most 0.01 GiB, agents lower PPL by improving the quantizer or redistributing precision (§ 4.1). Complete-model feedback changes selection. Reward feedback reverses 17 of 79 prior local rankings, leading agents to revert candidates or revise their local validation (§ 4.2). Likelihood alone does not validate a model. Intermediate models gain and lose correct answers at nearly unchanged PPL, and consistency tests expose inference bugs that likelihood misses (§ 4.4).

Compression as a Terminal Task

A compression task asks an agent to produce a model that approximates a source LLM under a specified storage allowance. Its output includes both the numerical representation and the code that runs it. The task couples model approximation with implementation and experimental search.

Task Specification and Search Space

A task specifies a source model pp with its tokenizer, a storage limit CC, a hardware allocation, a pinned inference runtime, and a wall-clock budget TT. The agent works in an offline container with read-only source weights, starter inference code, searchable compression papers, calibration and recovery data, and a separate set of questions for local testing. The runtime provides scheduling and uncompressed operations. The agent chooses its compression algorithm, produces the weights, and implements the operations required by its chosen representation.

Let a=(W,I)a=(W,I) denote a candidate, where WW contains compressed weights and associated metadata and II is its inference implementation. Together they define the executable model qaq_a. The search space includes quantization formats and rounding rules, component-specific precision, pruning, low-rank representations, and distillation. Changes to this representation can also require changes to II. The task does not prescribe an order for making these decisions.

The storage allowance constrains two quantities. Spkg(a)S_{\mathrm{pkg}}(a) counts the saved model package, including code and tokenizer assets. Sres(a)S_{\mathrm{res}}(a) counts model storage retained on the GPU during inference, including scales, codebooks, padding, and persistent dequantized weights. Shared allocations count once. KV caches, recurrent states, and temporary computation workspace are excluded. These rules prevent a small saved checkpoint from satisfying the task by retaining a full-precision model during execution. The allowance constrains resident weights rather than total peak GPU memory.

Experiments and Feedback

The agent allocates TT among implementation, calibration, recovery, local tests, and evaluation of complete candidates. It can use local measurements such as reconstruction error, KL divergence, or negative log-likelihood to choose what to build. A supplied evaluator returns PPL-based reward, package and resident model storage, decode timing, and execution diagnostics. All such calls consume the same wall-clock budget as model construction. The agent must judge whether another evaluation is worth its cost.

This interaction follows the budgeted experimentation setting of MLAgentBench and RE-Bench ([17] (Wijk et al. 2025)). Candidate versions and experiments are recorded so that the final model can be connected to the revisions that produced it. At time TT, the submitted candidate is frozen. A separate container performs final verification, adding held-out generated-answer tests and implementation review to the likelihood and storage measurements.

Likelihood Objective

Let DsD_s and DvD_v denote the search and verification datasets. Both contain continuations sampled from the source model, using disjoint contexts. A dataset D={(hn,yn)}n=1ND=\{(h_n,y_n)\}_{n=1}^{N} records each scored token yny_n and its context and preceding continuation hnh_n. Source and compressed models score the same NN continuation positions. Their empirical cross-entropy and perplexity are

H^D(q)=−1N∑n=1Nlog⁡q(yn∣hn),(1)\widehat{H}_{\mathcal{D}}(q)=-\frac{1}{N}\sum_{n=1}^{N}\log q(y_n\mid h_n), \tag*{(1)}
PPL⁡D(q)=exp⁡(H^D(q)).\operatorname{PPL}_{\mathcal{D}}(q)=\exp(\widehat{H}_{\mathcal{D}}(q)).

For candidate a=(W,I)a=(W,I) with executable model qaq_a, the empirical excess cross-entropy is

Δ^D(a)=H^D(qa)−H^D(p).(2)\widehat{\Delta}_{\mathcal{D}}(a)=\widehat{H}_{\mathcal{D}}(q_a)-\widehat{H}_{\mathcal{D}}(p). \tag*{(2)}

Under source sampling, its expectation equals the next-token KL divergence averaged over scored histories:

E[Δ^D(a)]=Eh∼μp[DKL(p(⋅∣h)∥qa(⋅∣h))].(3)\mathbb{E}\left[\widehat{\Delta}_{\mathcal{D}}(a)\right]=\mathbb{E}_{h\sim\mu_p}\left[D_{\mathrm{KL}}\left(p(\cdot\mid h)\mathbin{\|}q_a(\cdot\mid h)\right)\right]. \tag*{(3)}

Here μp\mu_p is the distribution of scored histories induced by the context mixture and source continuations. This objective measures likelihood preservation on source histories; answer tests assess the candidate’s autoregressive behavior.

The task minimizes the search estimate under separate saved and resident model-storage constraints:

minimize⁡a=(W,I)Δ^Ds(a)subject to⁡Spkg(a)≤C,Sres(a)≤C.(4)\begin{aligned} \underset{a=(W,I)}{\operatorname{minimize}}\quad&\widehat{\Delta}_{\mathcal{D}_s}(a)\\ \operatorname{subject\ to}\quad&S_{\mathrm{pkg}}(a)\le C,\\ &S_{\mathrm{res}}(a)\le C. \tag*{(4)} \end{aligned}

Construction, local tests, and supplied evaluations share the wall-clock budget TT. The evaluator returns the source-to-compressed PPL ratio:

FD(a)=PPL⁡D(p)PPL⁡D(qa)=exp⁡(−Δ^D(a)).(5)\begin{aligned} F_{\mathcal{D}}(a)&=\frac{\operatorname{PPL}_{\mathcal{D}}(p)}{\operatorname{PPL}_{\mathcal{D}}(q_a)}\\ &=\exp\left(-\widehat{\Delta}_{\mathcal{D}}(a)\right). \tag*{(5)} \end{aligned}

Equal measured likelihood gives FD(a)=1F_{\mathcal{D}}(a)=1. During search, let U(a)=1U(a)=1 when execution succeeds and both storage limits hold. The feedback reward is

rs(a)={10FDs(a),U(a)=1,0,U(a)=0.(6)r_s(a)= \begin{cases} 10F_{\mathcal{D}_s}(a), & U(a)=1,\\ 0, & U(a)=0. \end{cases} \tag*{(6)}

Timing is returned separately and does not enter this reward. In trajectory plots, EE abbreviates the measured excess cross-entropy Δ^D(a)\widehat{\Delta}_{\mathcal{D}}(a), and FF denotes the corresponding PPL ratio when the dataset is clear.

Independent Acceptance Tests

Final verification evaluates the frozen weights through the submitted inference code and official tokenizer. Its contexts are disjoint from those used for search-time reward. The evaluator controls inputs and measurements and checks four aspects of the artifact.

The evaluator computes FDvF_{\mathcal{D}_{\mathrm{v}}} on verification continuations and measures both package and resident model storage. Retained high-precision copies count at their actual size. Storage is measured after loading and after inference to include allocations that persist once the model has run.

Let B\mathcal{B} contain GSM8K, HumanEval, and TriviaQA, with Ab(a)A_b(a) denoting accuracy on benchmark bb. Compressed and source models receive the same questions and generation settings. Mean accuracy and the acceptance condition are

A(a)=1∣B∣∑b∈BAb(a),(7)A(a)=\frac{1}{|\mathcal{B}|}\sum_{b\in\mathcal{B}}A_b(a), \tag*{(7)}
A(a)≥ρA(p),ρ=0.8,A(a)\ge\rho A(p),\qquad\rho=0.8,

where A(p)A(p) is the BF16 reference fixed before search. The threshold is set before search and withheld from agents. Held-out answer scores are absent from search-time reward. Separate questions support the agents’ local generation tests.

Automated checks and independent review assess compliance with the construction and inference rules. Tokens and probabilities must come from the submitted forward pass and packaged weights, with unscaled logits for likelihood scoring. Review checks model-storage accounting, external weights, hard-coded outputs, and manipulation of probability or timing measurements.

A separate test measures elapsed time LL between the first and last of 128 streamed tokens after warmup. We report batch-one decode speed vdec=127/Lv_{\mathrm{dec}} = 127/L tokens/s: 128 tokens span 127 inter-token intervals. Prefill and time to first token are excluded. Speed does not enter the reward, distinguishing compression quality from execution efficiency.

Writing V(a)=1V(a)=1 when execution completes, the storage limits and answer threshold hold, and integrity checks pass, the final normalized score is

R(a)={FDv(a),V(a)=1,0,V(a)=0.(8)R(a) = \begin{cases} F_{\mathcal{D}_v}(a), & V(a)=1,\\ 0, & V(a)=0. \end{cases} \tag*{(8)}

The final score uses verification data and the full acceptance tests; search feedback uses Ds\mathcal{D}_s and U(a)U(a). Thus candidate selection and final acceptance evaluate distinct requirements with separate data.

Empirical Study

We first ask whether agents can deliver useful models under the task’s constraints. We then examine the recorded searches to understand how agents improve candidates and which failures survive to submission.

Experimental Setting

We study text generation from two mixture-of-experts (MoE) checkpoints, Nemotron-3.5 Super VL 120B-A12B and Qwen3.5 122B-A10B, and the dense Qwen3.8 27B checkpoint. Each MoE model has 32 and 48 GiB tasks; the dense model has 8 and 12 GiB tasks. Pairing limits holds the source model fixed while changing the need for aggressive compression. Pilot measurements on Qwen3.8 guide these allowances. Larger pilot limits admit four-bit GPTQ with group-size or damping refinements; tighter limits make low-bit representations and precision allocation more consequential. Some pilot models retain BF16 weights during inference despite small packages, motivating separate limits on saved and resident model storage (Appendix § 12).

Claude Sonnet 5.5, Claude Opus 5.5, GLM 5.3, Kimi K3.1 Pro, MiniMax M3.1 Flash, and DeepSeek V4.1 Flash 1017 each attempt all six tasks. Every run uses Claude Code with maximum reasoning effort, one H200 NVL, vLLM 0.32.0, and an eight-hour budget. Agents receive papers and FineWeb, UltraChat-200k, and WikiText-103 for calibration or recovery ([8] (Penedo et al. 2024; Merity et al. 2017)). MATH-500 provides separate questions for local answer tests ((Lightman et al., 2024)). There is one run per agent–task pair.

A uniform GPTQ baseline uses groups of 128 weights and 128 calibration sequences of 2,048 tokens ([12]). It chooses the largest uniform bit width that fits the package limit and adjusts embedding and output-head precision to the remaining space. We also evaluate released quantizations of the same checkpoints: Unsloth Dynamic GGUF for Nemotron and Qwen3.5, and GSQ-RC0 for Qwen3.8 ((Dadgarinejad et al., 2026; Helgic and Alistarh, 2026)).

All compressed models run on one GPU. Agent artifacts and GPTQ use vLLM ((Kwon et al., 2023)) with packed weights; published weights run unchanged in their native packed GGUF representation using llama.cpp. BF16 uses two GPUs.

Figure 3. Verification measurements. (a) Source pp and candidate qq score the same source-generated tokens. (b) Package and resident model storage must each fit CC, including persistent reconstructed weights. (c) Mean accuracy over three benchmarks must retain 80% of BF16 accuracy, using fifty questions each. (d) Batch-one decode speed is 127/LL tokens/s.

for the MoE models and one for the dense model. All comparisons share the measurement protocol; TPS also reflects the submitted kernel implementations and dequantization overhead.

Search-time and verification likelihood each use 256 contexts of 512 tokens from the Dolma 3 Dolmino mixture (Team Olmo et al. (2025)). The sets use disjoint documents with matched source proportions. For each context, BF16 samples 1,536 tokens at temperature one without probability truncation; both models score those continuations. We report final PPL in tables and search-time feedback in trajectory plots. Cross-model comparisons use PPL increase over BF16.

Answer accuracy is the mean over fifty held-out questions each from GSM8K, HumanEval, and TriviaQA (Cobbe et al., [6]; Chen et al., [5]; Joshi et al., 2017}), with identical questions and greedy decoding for source and compressed models. Acceptance uses BF16 references fixed before search. The main table also reports fresh BF16 measurements; these remeasurements supply the 77.33% versus 79.56% aggregate comparison below.

Verification runs on an idle GPU and counts model allocations after loading and inference. Decode uses batch size one, a 2,048-token prompt, and 128 output tokens. After warmup, LL is the median of three round means over three requests each, timed between first and last token. Reported decode speed is 127/L127/L tokens/s. The fixed BF16 references, generation budgets, measurement protocol, per-artifact results, and additional experiments appear in Appendices § 9, § 11 and § 13.

Delivered Models

Twenty-nine of the thirty-six submissions pass final verification (Table 1). All eighteen at the looser limits pass. At the tighter limits, six fail the answer-accuracy requirement and one times out during answer generation. Selecting the lowest-PPL agent artifact for each task gives an average 3.4% PPL increase over BF16. Those six artifacts average 77.33% downstream accuracy, compared with 79.56% for the remeasured BF16 models.

Method / agentNemotron-3.5 120B-A12B / 32 GiB PPL↓\downarrowNemotron-3.5 120B-A12B / 32 GiB Acc.↑\uparrowNemotron-3.5 120B-A12B / 32 GiB TPS↑\uparrowQwen3.5 122B-A10B / 32 GiB PPL↓\downarrowQwen3.5 122B-A10B / 32 GiB Acc.↑\uparrowQwen3.5 122B-A10B / 32 GiB TPS↑\uparrowQwen3.8 27B / 8 GiB PPL↓\downarrowQwen3.8 27B / 8 GiB Acc.↑\uparrowQwen3.8 27B / 8 GiB TPS↑\uparrow
BF161.68880.00147.52.75584.67158.33.22174.0061.4
Released model———2.90083.33122.93.76465.3375.5
GPTQ4.6805.3320.97.1856.0026.47.4804.6712.6
Sonnet 5.51.80176.6757.22.88378.6739.83.36368.6723.4
Opus 5.51.81185.3348.63.00376.6773.23.37066.0026.2
GLM 5.31.87174.6716.6Not passedNot passedNot passedNot passed
Kimi K3.1Not passedNot passed3.03578.0043.1Not passedNot passed
MiniMax M3.12.08475.3311.4Not passedNot passedNot passedNot passed
DeepSeek V4.1Not passedNot passed3.11078.0012.74.28361.3313.7
Method / agentNemotron-3.5 120B-A12B / 48 GiB PPL↓\downarrowNemotron-3.5 120B-A12B / 48 GiB Acc.↑\uparrowNemotron-3.5 120B-A12B / 48 GiB TPS↑\uparrowQwen3.5 122B-A10B / 48 GiB PPL↓\downarrowQwen3.5 122B-A10B / 48 GiB Acc.↑\uparrowQwen3.5 122B-A10B / 48 GiB TPS↑\uparrowQwen3.8 27B / 12 GiB PPL↓\downarrowQwen3.8 27B / 12 GiB Acc.↑\uparrowQwen3.8 27B / 12 GiB TPS↑\uparrow
BF161.68880.00147.52.75584.67158.33.22174.0061.4
Released model1.76482.67100.52.80882.00106.93.33466.0074.6
GPTQ1.87576.0016.02.98779.3320.23.54646.0011.1
Sonnet 5.51.72882.0047.42.78885.3346.33.25376.6716.7
Opus 5.51.72678.0068.22.81882.6752.93.27278.6731.4
GLM 5.31.77878.6740.32.81482.6710.53.38380.6710.4
Kimi K3.11.72982.0013.62.83584.677.83.33374.678.0
MiniMax M3.11.75281.339.12.81484.6727.63.39976.6710.9
DeepSeek V4.11.74979.3316.62.80384.6710.83.39972.0018.4

Table 1. Verified models under tighter (a) and looser (b) storage limits. Perplexity (PPL) is measured on held-out BF16-generated continuations; accuracy (%) averages GSM8K, HumanEval, and TriviaQA. TPS is decode tokens per second at batch size one, computed as 127/L127/L from the 128-token first-to-last interval LL. Note that TPS is excluded from the PPL-based reward, and weight unpacking and dequantization can offset the benefits of reduced memory traffic. Compressed models use one H200 NVL; BF16 uses one for the dense model and two for the MoE models. Released models use Unsloth Dynamic UD-IQ2_XXS (Nemotron), Unsloth Dynamic (Qwen3.5), and GSQ-RCO (Qwen3.8). Complete measurements, including rejected artifacts, are in Table A3.

The storage allowance influences every agent’s result, but does not fully explain quality. Each agent’s larger-allowance artifact has lower PPL than its tighter counterpart. Yet Sonnet’s 8 GiB dense model also has lower PPL than three agents’ 12 GiB models. A better representation can therefore outweigh additional storage in these comparisons. Uniform GPTQ passes the answer threshold on the looser MoE tasks and fails on the remaining settings, indicating that a fixed recipe covers only part of the task suite.

Released quantizations provide a stronger comparison. The best agent artifact by PPL improves on each of the five measured released-model references. Answer accuracy distinguishes designs further: the tighter Qwen3.5 release has higher accuracy than the lowest-PPL agent model, whereas Sonnet improves both PPL and accuracy over the released dense references. PPL and answer accuracy thus affect which artifact would be preferred.

Figure 1 places the results alongside encoded size and API cost. Sonnet has the best average PPL rank, followed by Opus; the largest API expenditure does not correspond to the best rank. The remaining analysis follows the concrete decisions behind these outcomes, rather than using spending or activity alone to explain them.

Figure 1. Agent-driven compression outcomes. (a) Mean GSM8K, HumanEval, and TriviaQA accuracy relative to BF16 versus package bits per original parameter; values below 80% lie on the axis. (b) API cost versus within-task PPL rank; diamonds average six tasks per agent. Pareto frontiers use passing models for the accuracy–size comparison in (a).

How Agents Improve Compression

Agents can change compression algorithms, storage allocation across model components, or candidate evaluation. Recorded trajectories connect these choices to model measurements and subsequent actions. Within-task comparisons follow the recorded revisions; a paired ablation isolates a quantizer decision after search.

Representation and Storage Tradeoffs

The first question is where an agent finds improvement once a model fits the storage limit. It can allocate more bytes to a sensitive component only by saving them elsewhere, or improve the approximation represented by existing codes. In 24 recorded revisions whose package sizes differ by at most 0.01 GiB, 21 lower PPL measured by reward feedback. These pairs span all six agents and all six tasks (Figure 4). These choices improve the quantizer or redistribute precision within a feasible model.

Figure 4. Revisions within recorded searches. Thirty-five pairs from 21 runs compare candidate PPL through reward feedback. Excess cross-entropy is E=ln⁡(PPL/PPLBF16)E = \ln(\mathrm{PPL} / \mathrm{PPL}_{\mathrm{BF16}}); the vertical axis is 100(Eafter/Ebefore−1)100(E_{\mathrm{after}}/E_{\mathrm{before}} - 1). (a) Changes by design factor, with lower-PPL/total counts. (b) The same pairs by package-size change; the band denotes changes within 0.01 GiB, and crosses mark candidates subsequently reverted.

On tighter Qwen3.5, Kimi changes clipping and GPTQ damping while keeping package size unchanged, reducing feedback PPL from 3.738 to 3.132. Across the trajectories, level and clipping revisions improve PPL for three agents on five tasks. Bit width fixes the code count; clipping and rounding determine how those codes approximate weights and activations ([12] (Shao et al. 2024)). The paired dense-model ablation in Appendix § 13.§ 2 keeps precision allocation, calibration, and inference implementation fixed while replacing GLM’s MSE-selected clipping ranges with min–max ranges. The selected ranges give lower verification PPL and higher answer accuracy. Both variants occupy the same saved and resident model storage. All three answer benchmarks favor the selected ranges. This connects the agent’s quantizer choice to the quality of the complete model after search.

Sensitivity guides mixed-precision bit allocation [9], but agents also allocate storage to metadata. Kimi reduces dense-model group-scale storage to fit additional four-bit weights. DeepSeek raises non-expert precision on Nemotron, while Opus gives later Qwen3.5 layers more expert precision. Weights and metadata share the package budget. Kimi reduces the storage used for scales to make room for higher weight precision; DeepSeek and Opus change where that precision is assigned. These examples show why the allocation includes quantizer overhead as well as the nominal bit width of the weights.

An agent can also remove parameters to afford higher precision for those that remain. GLM and MiniMax test additional expert or MLP-width removal; their recorded comparisons favor the less-pruned alternatives. MiniMax nevertheless retains a large dense-model width reduction after rejecting an early two-bit implementation. Its subsequent generation tests expose a substantial loss. MiniMax’s decision to retain pruning links its precision choice to parameter removal. Subsequent numerical refinements operate on a model with fewer parameters.

Local Metrics and Candidate Selection

The second question is how an agent chooses which design to retain. Local experiments guide which candidate quantizers and precision allocations an agent builds. We examine what happens when reward feedback disagrees with that local ranking: whether the agent reverts the candidate, revises its numerical choices, or changes the validation used in subsequent experiments. Across 79 matched comparisons from 26 runs, reward feedback reverses 17 prior local rankings across all six tasks and all agents except MiniMax. Eight reversals lead to explicit rollback: five by Kimi and three by DeepSeek. Across the eligible comparisons, agents also refine calibration and quantization or reallocate precision Appendix § 13.§ 3. Model-level measurements thus influence both the candidate an agent retains and the numerical choices it revisits.

A controlled diagnostic on dense Qwen3.8 varies levels, clipping, rotation, rounding, and attention–MLP bit allocation at similar encoded size Figure 5. Forward passes use BF16 reconstructions, separating numerical representation from packed-kernel execution. A k-means codebook improves weight reconstruction MSE over clipped uniform levels but increases PPL. GPTQ rounding has the opposite pattern: larger weight MSE and lower PPL. Calibration-activation output error follows PPL more closely in this comparison. Weight MSE treats reconstruction errors equally across input directions. Linear-output MSE weights those errors by the calibration activations. A small reconstruction error in a direction with large activation magnitudes can affect the layer’s output more than a larger error in a weakly activated direction. Complete-model PPL measures the resulting behavior after all quantized layers are composed. These criteria can therefore favor different revisions at similar encoded size.

Figure 5. Local error and model likelihood. Twelve Qwen3.8 variants use approximately 3.14 encoded bits per quantized decoder weight, with BF16 embeddings and output head. PPL is measured using BF16 reconstructions on the source continuations used for reward feedback. (a) Weight reconstruction MSE. (b) Linear-output MSE on BF16 FineWeb calibration activations. Each error is normalized by its BF16 squared norm; the dotted line denotes BF16 PPL.

On looser Qwen3.5, Opus initially favors more precision for early layers. It revises local validation to score model-generated continuations, then favors later layers, an allocation supported by reward feedback. On the tighter task, it spends 1.7 hours exploring a one-bit expert tier. Local likelihood favors the new candidate, but feedback favors the previous allocation; Opus restores and refines that allocation. Conflicting measurements lead it to revise local validation in one search and revert the candidate in the other.

Later evaluations increasingly match a run’s best PPL Figure 6. Some confirm that a candidate is worth keeping; others follow successful numerical revisions. Kimi improves its dense model after adding a low-rank residual and distillation. MiniMax instead repeatedly evaluates a pruned candidate after its last improvement. Reward feedback reports a tested candidate’s performance; the agent then decides whether to refine its quantizer, change precision allocation, or reconsider the representation.

Figure 6. Feedback during candidate selection. (a) Best-so-far PPL for four searches; ticks indicate whether complete-model feedback confirms or reverses a local ranking. (b) Outcomes of 301 reward evaluations across 36 runs, grouped by search time. Rewards within 0.002 of the best are treated as the same PPL. (c) The next outcome after 46 worse candidates. Reward equals ten times the source-to-compressed PPL ratio; all plotted evaluations occur during search.

Validating Generation and Execution

Validating Generation and Execution

A prior pruning study reports mismatches between perplexity and downstream performance (Khanal and Capone, 2024). For agents, detecting a generation regression and finding a feasible repair are distinct challenges. PPL scores fixed source-generated histories, while generation feeds the candidate’s own choices back into the model. Repetition, early termination, and changed problem solving therefore require direct checks. Intermediate verification tracks these changes between versions; trajectories show whether agents detect the loss and find a repair they can retain.

We verify retained intermediate models after search using the same held-out inputs as the final artifacts. In Opus’s tighter dense-model run, allocation and distillation improve PPL while GSM8K accuracy falls from 88% to 72% across trained versions. HumanEval and TriviaQA improve over the same versions. Twelve comparisons from four runs show both newly correct and newly incorrect answers after small PPL changes (Figure 7). A single scalar can conceal changes in which capabilities survive.

Figure 7. Capability changes across candidate models. (a) Verification PPL and answer accuracies for Opus on Qwen3.8 / 8 GiB; shading marks distillation and dashed lines mark BF16. (b) Answers gained and lost across twelve successive-version pairs from four runs, plotted against PPL change. (c) HumanEval failure types for selected rejected artifacts and their BF16 references. Each benchmark has fifty questions.

Generation failures also differ in kind. Pruned GLM and MiniMax models produce repeated or truncated HumanEval completions, while most BF16 errors are executable programs that fail tests. This matters for acceptance: an artifact may preserve scored-token likelihood yet become unreliable at completing an answer.

MiniMax’s local reasoning tests show a large loss after MLP-width pruning, relative to its unpruned three-bit reference. GLM’s expert-pruned Qwen3.5 candidate similarly underperforms an eight-bit reference on MATH-500. Both agents test alternatives and retain pruning without finding a feasible design that resolves the loss. Opus tests generation during distillation but makes later PPL-oriented changes without repeating the same generation test on its last versions. Those local MATH-500 measurements and the post-search GSM8K loss are separate tests. The trajectory ries expose two distinct requirements: finding a candidate that fixes a known problem and keeping tests current with the artifact ultimately submitted.

Figure 8. Testing and acting on failures. MiniMax’s dense-model run (a) and GLM’s Qwen3.5 run (b) retain pruning; Opus continues dense-model distillation (c); MiniMax repairs expert indexing on Nemotron (d). Curves show feedback PPL. Diamonds show the agents’ generation tests relative to an unpruned three-bit reference (a), an eight-bit reference (b), or BF16 (c). Final markers show held-out benchmark accuracy relative to BF16.

On Nemotron, MiniMax compares token probabilities under batched and per-expert inference. The disagreement identifies an expert-indexing bug. Repair reduces the feedback PPL increase over BF16 from about 11% to 4.5%, and the final model passes verification. Here, the agent repairs execution while keeping the representation fixed. In the pruning cases, generation tests reveal regressions, but agents retain the affected designs. Detecting a failure and finding a feasible change to the compression design or implementation that resolves it are distinct challenges.

Two Complete Compression Strategies

The dense and MoE case studies in Figure 9 bring these decisions together. Both retain active parameters at the looser limits, but their numerical methods lead to different allocations and execution paths.

Figure 9. From numerical design to execution. Sonnet compresses dense Qwen3.8 to 12 GiB using rotation, trellis codes, and sensitivity-based allocation (a). Opus compresses MoE Qwen3.5 to 48 GiB using rotated GPTQ and expert-specific precision (b). Rows connect the offline pipeline, component precision, inference operations, and candidate versions. Curves show search-time reward; boxes list explored alternatives that were discarded.

  • Sonnet: trellis codes and bit allocation. Sonnet replaces scalar GPTQ with Hadamard-rotated trellis codes, error compensation, and Viterbi search [12] (Tseng et al. 2024b). Matrix experiments help choose the representation. KL ablations supply a fitted loss model for bit allocation; token frequency guides embedding and output-head precision. Representation and allocation changes produce the largest reward gain, followed by calibration refinements. Sonnet rejects Fisher-weighted allocation when its measured KL worsens and keeps a smaller trellis

when a larger one offers little reconstruction gain at greater search cost. Its final artifact has PPL 3.253 and 76.67% mean answer accuracy, versus BF16’s 3.221 and 74.00%.

  • Opus: expert-specific precision. Opus uses rotated GPTQ with different bit widths across experts. Revising validation shifts the favored allocation toward later layers, while natural text remains the calibration source. Its search rejects additional precision tiers and alternative rotations. The final artifact has PPL 2.818 and 82.67% accuracy, versus BF16’s 2.755 and 84.67%.

Intermediate verification separates likelihood from answer preservation. Sonnet’s versions v3, v4, and v5 attain PPL 3.2572, 3.2531, and 3.2525, but answer 111, 104, and 115 of 150 questions correctly. Opus’s Qwen3.5 version v4 has PPL 2.8172 and 120 correct answers; v5–v8 have PPL 2.8178 and 124 correct answers. These tests use final verification inputs; search-time feedback guides candidate selection. Nearly unchanged likelihood thus accompanies different generated answers.

Sonnet reconstructs weight tiles from its stored trellis codes as they are needed for matrix computation. Other designs can first reconstruct a whole weight matrix in temporary workspace and then use dense operations. Both approaches can satisfy the resident-storage limit because the reconstructed values are temporary. They expose different amounts of unpacking work and memory traffic to inference, which contribute to the measured execution cost. In the dense looser task, Sonnet improves PPL and accuracy over the released GSQ-RCO artifact, but decodes at 16.7 tokens/s versus the release’s 74.6 tokens/s at batch size one (Table A3). This illustrates why the measured storage–quality improvement does not by itself establish a speedup, and why representation and kernels must be considered together (Lin et al., 2024, 2025).

MethodPPL↓Acc.↑PackageResidentTPS↑Gate
Nemotron-3.5 120B-A12B, 32 GiB
GPTQ4.6805.3331.95931.98020.9Fail
Sonnet1.80176.6731.66531.64957.2Pass
Opus1.81185.3331.94331.91248.6Pass
GLM1.87174.6731.87331.84116.6Pass
Kimi2.05158.6731.99931.97512.5Fail
MiniMax2.08475.3330.93030.90511.4Pass
DeepSeek2.36963.3331.84431.81413.8Fail
Nemotron-3.5 120B-A12B, 48 GiB (release: UD-IQ2_XXS)
GPTQ1.87576.0045.98145.99516.0Pass
Unsloth1.76482.6747.62544.685100.5Pass
Sonnet1.72882.0047.85047.83447.4Pass
Opus1.72678.0047.90447.85468.2Pass
GLM1.77878.6747.23247.20740.3Pass
Kimi1.72982.0047.95847.91513.6Pass
MiniMax1.75281.3347.68647.6709.1Pass
DeepSeek1.74979.3347.68947.67316.6Pass
Qwen3.5 122B-A10B, 32 GiB (release: UD-IQ1_M)
GPTQ7.1856.0031.60431.64826.4Fail
Unsloth2.90083.3331.88331.861122.9Pass
Sonnet2.88378.6731.74631.76539.8Pass
Opus3.00376.6731.72831.75073.2Pass
GLM3.63244.6731.71531.73418.3Fail
Kimi3.03578.0031.91831.93743.1Pass
MiniMax3.273—31.510——Timeout
DeepSeek3.11078.0031.84031.83212.7Pass
Qwen3.5 122B-A10B, 48 GiB (release: UD-IQ3_S)
GPTQ2.98779.3347.13347.17320.2Pass
Unsloth2.80882.0043.37243.349106.9Pass
Sonnet2.78885.3347.66947.68946.3Pass
Opus2.81882.6747.84747.86652.9Pass
GLM2.81482.6747.68847.70710.5Pass
Kimi2.83584.6747.97647.9957.8Pass
MiniMax2.81484.6747.65847.77027.6Pass
DeepSeek2.80384.6747.68647.70610.8Pass
Qwen3.8 27B, 8 GiB (release: IQ2_XS)
GPTQ7.4804.677.3547.51512.6Fail
GSQ-RCO3.76465.337.8567.83475.5Pass
Sonnet3.36368.677.9577.95023.4Pass
Opus3.37066.007.9777.99526.2Pass
GLM6.06123.337.9477.95313.7Fail
Kimi4.17654.007.9737.9927.9Fail
MiniMax8.1639.337.9737.84015.1Fail
DeepSeek4.28361.337.9347.95013.7Pass
Qwen3.8 27B, 12 GiB (release: IQ3_S)
GPTQ3.54646.0011.40111.53111.1Fail
GSQ-RCO3.33466.0010.97510.95374.6Pass
Sonnet3.25376.6711.95611.97716.7Pass
Opus3.27278.6711.98011.97031.4Pass
GLM3.38380.6711.95411.97310.4Pass
Kimi3.33374.6711.96711.9868.0Pass
MiniMax3.39976.6711.77911.79810.9Pass
DeepSeek3.39972.0011.94811.96718.4Pass

Table A3. Complete final-verification measurements by source model and storage limit. PPL scores held-out BF16-generated continuations; accuracy (%) averages GSM8K, HumanEval, and TriviaQA. Package and resident model storage are in GiB. TPS is batch-one decode speed, 127/LL for a 128-token first-to-last interval LL. The accuracy gate requires at least 80% of the fixed BF16 reference.

Related Work

Agentic Benchmarks

AlphaEvolve optimizes algorithms and LLM-training infrastructure with coding agents (Novikov et al., 2025). The Darwin Gödel Machine modifies agent code and evaluates the resulting coding agents (Zhang et al., 2026). We study a complementary efficiency objective: compressed model weights and their execution under a resource budget. The terminal task connects numerical choices and validation decisions to the quality of the executable model.

Terminal-Bench evaluates agents on verified command-line tasks (Merrill et al., 2026); Harbor provides a container-based framework for executing tasks and evaluating agents [13]. SWE-bench studies repository repair [18], and AgentBench evaluates interactive environments (Liu et al., 2024). MLAgentBench, MLE-bench, and RE-Bench examine ML experimentation, competition solutions, and research engineering [17, 3] (Wijk et al. 2025). KernelBench tests correctness and runtime for specified tensor computations (Ouyang et al., 2025); ServeLearnBench studies learning from serving feedback (Zheng et al., 2026). Compression adds the choice of numerical approximation: the agent must select, implement, and validate a model whose generated answers remain useful.

LLM Compression

GPTQ, AWQ, SmoothQuant, and OmniQuant use calibration to choose rounding, scaling, or clipping [12] (Lin et al. 2024; Xiao et al. 2023; Shao et al. 2024). QuIP, QuaRot, and SpinQuant improve quantization through incoherence processing or rotations [4, 2] (Liu et al. 2025); QuIP#, AQLM, and QTIP develop lattice, additive, or trellis representations (Tseng et al., 2024a; Egiazarian et al., 2024; Tseng et al., 2024b). HAWQ-V2 uses sensitivity for mixed precision [9], while GSQ and RCO optimize quantization and budget allocation [7] (Helgic and Alistarh 2026). Pruning removes weights or dimensions; low-rank methods reduce matrix rank [1] (Frantar and Alistarh 2023; Sun et al. 2024; Wang et al. 2025). Recovery can use quantization-aware fine-tuning (Malinovskii et al., 2024) or distillation [15] (Muralidharan et al. 2024). Serving systems optimize memory management and attention execution (Kwon et al., 2023; Ye et al., 2025); TinyChat and QServe couple quantized formats with matching kernels (Lin et al., 2024, 2025).

These methods supply components; we study agents that select, combine, implement, and validate a complete compressed model.

Conclusion

LLM compression provides a concrete task for studying agent-driven model improvement. Across 36 runs, 29 artifacts pass independent verification. Recorded searches show quality gains at nearly fixed storage, local rankings overturned by complete-model feedback, and generation or implementation failures missed by likelihood alone. Effective compression requires numerical design, experimental judgment, and validation of the final executable model. Extending this step toward RSI requires testing whether a compressed model retains the ability to carry out subsequent compression tasks.

Acknowledgements

This work was partially supported by Google Research Award, Google ML & System Junior Faculty Award, Amazon Research Award, Fireworks AI, Intel, Li Auto, Moffett AI, Daytona, and CMU CyLab Seed funding. This material is also based upon work supported by the National Science Foundation under Grant Nos. CCF-2504353 and CCF-2247014, and by IARPA. Any opinions, findings, conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation.

AI Usage Disclosure

Claude Code and OpenAI Codex assisted with manuscript rewriting, literature and citation checks, LaTeX preparation, and PDF compilation. The compression artifacts and numerical measurements come from the recorded experimental runs described in the paper.

Evaluation Protocol

Each likelihood set contains 256 contexts of 512 tokens from Dolma 3 Dolmino, stratified by source family. BF16 samples 1,536 continuation tokens at temperature one without probability truncation; both models score the 393,216 tokens. Reward feedback and verification use disjoint documents with matched source proportions. Table A1 gives the references fixed before agentic design.

SourceReward feedback PPLVerification PPLA(p)A(p) (%)
Nemotron-3.51.7926481.68801181.33
Qwen3.52.7680582.75550085.33
Qwen3.83.0974263.22074374.00

Table A1. BF16 references fixed before agentic design. PPL is perplexity on disjoint source-generated continuations for reward feedback and verification. A(p)A(p) is mean accuracy (%) on GSM8K, HumanEval, and TriviaQA, used to set the accuracy gate.

Source and compressed models use the same questions, greedy decoding, and settings in Table A2. Acceptance requires A(a)≥0.8A(p)A(a) \ge0.8A(p), with the fixed reference in Table A1.

DatasetShotsTokensJudgment
GSM8K4320Final number
HumanEval0384Unit tests
TriviaQA524Normalized aliases

Table A2. Multi-benchmark accuracy protocol. Each benchmark uses fifty held-out questions and greedy decoding. Shots gives in-context examples; Tokens gives maximum answer length. The three benchmark accuracies are averaged equally.

Package size includes weights, inference code, and tokenizer assets. Resident storage counts persistent weights, quantization metadata, and reconstructed copies after loading and inference. Shared allocations count once; KV caches, recurrent states, and temporary workspace are excluded.

Batch-one decode uses 2,048 input and 128 output tokens after warmup. TPS is 127/L127/L, where LL is the median of three round means, each over three first-to-last-token intervals. BF16 speeds are 147.5, 158.3, and 61.4 tokens/s for Nemotron, Qwen3.5, and Qwen3.8, respectively. Final verification has a 3,600-second timeout.

Candidate-ranking pairs match local rankings with subsequent reward feedback for the same configurations. Package bits per original parameter equal eight times package bytes divided by parameter count; component precision uses resident bytes and component parameter count. API cost uses recorded input, output, and cached tokens at October 26, 2026 OpenRouter rates. Average ranks weight all six tasks equally.

Search Activity and Stopping

We examine how agents use the design budget and when they stop revising weights (Figure A1). Agents were instructed to carry out agentic design end to end until the time limit. Runs proceeded without human intervention or follow-up prompts.

Computation and stopping during agentic design: (a) More GPU, lower PPL; (b) Early stop: a weak link; (c) More calls, no gain; (d) Budget left at first stop

Figure A1. Computation and stopping during agentic design. (a–c) Final-verification perplexity (PPL) increase over the best submission on the same task versus GPU-busy share of the run, first completion declaration as a fraction of the design budget, and tool calls divided by the task median. Values above 30% lie at the upper boundary. (d) Budget remaining at the first completion declaration for each of thirty-six runs. Filled points mark subsequent weight changes; counts give changed/total runs, and bars show medians. Agents often revise weights after declaring completion.

Seventeen runs change weights after the agent first declares the task complete: five Kimi runs, four each for Opus and DeepSeek, two each for GLM and MiniMax, and none for Sonnet. Kimi has the earliest median first completion declaration and usually continues revising its weights. Median GPU-busy shares range from 30% to 60% across agents. These records distinguish a declaration of completion from the last effective model change.

Final and Intermediate Model Measurements

Table A3 contains the full final measurements, including models rejected by the answer threshold and decode TPS for both compression baselines.

Intermediate models are evaluated after search on the same verification inputs as the final submissions. The main-paper trajectory curves report reward feedback gathered during agentic design.

On Qwen3.8 at 8 GiB, Opus’s reallocation lowers verification PPL increase from 6.17% to 5.89%. Distillation lowers it to 5.00%, then further allocation and training reach 4.64%. GSM8K accuracy falls from 88% to 72% across trained versions, while the other benchmark changes are smaller.

Storage Limits and Reference Quantizations

Pilot compression runs on Qwen3.8 27B inform the task allowances and storage accounting. Pilot PPL is measured on 256 held-out C4 (Raffel et al., 2020) contexts of 2,048 tokens, with BF16 PPL 9.9227. The main study instead evaluates source-generated continuations.

Table A4 compares compression choices across four storage allowances. Pilot timing follows the first-to-last-token protocol in Appendix § 9 with an earlier vLLM release.

SubmissionPackage (GiB)PPL above BF16TPS↑ (bsz=1)
32 GiB
GLM: 4-bit GPTQ/Marlin17.34+3.09%106.1
GLM: 4-bit GPTQ (no example)17.34+3.22%106.0
Sol: optimized GPTQ16.43+3.38%111.9
GLM: 5-bit, group 6420.52+0.60%94.8
16 GiB
4-bit, group 128, damping 0.0115.00+3.19%109.0
4-bit, group 6415.35+2.58%108.6
4-bit, group 32 (over limit)16.06+1.98%107.6
4-bit, group 128, damping 0.115.00+2.79%109.3
GLM A: group 64 for MLP15.83+3.89%109.9
GLM B: 4-bit embedding/head14.99+3.10%109.1
12 GiB
3-bit, group 12811.18+13.32%110.6
Mixed 3/4-bit, group 12811.98+10.53%110.8
GLM A: 3/4-bit (load failure)11.83––
GLM B: Marlin repacking11.97+9.43%111.1
8 GiB
GLM: Hadamard, 2/3/4-bit7.96+1739%61.9
Qwen: 2/3/4-bit, clipping7.94+48.03%62.0
MiMo: 2/3/4-bit7.99+51.17%100.6
GLM: 2-bit, blockwise decode7.09+117.10%–
GLM: AutoRound/RTN7.83+94.27%–

Table A4. Pilot Qwen3.8 27B compression. PPL increase uses held-out C4 and BF16 PPL 9.9227. TPS is batch-one decode speed (127/LL for the 128-token first-to-last interval LL). GLM, Sol, Qwen, and MiMo denote GLM-5.3-Flash, GPT-6.1 Sol, Qwen3.8-Flash, and MiMo-V2.6-Pro, respectively.

At 16 GiB, finer groups lower PPL but increase metadata storage; the group-32 configuration exceeds the limit. At 12 GiB, mixed three- and four-bit weights improve on uniform three-bit quantization. At 8 GiB, similar package sizes yield different PPL. These results motivate paired limits that expose precision-allocation and representation decisions.

The pilot QAT Q2_0 release uses llama.cpp; the remaining releases in Table A5 use vLLM.

ReleasePackage (GiB)Resident (GiB)ΔPPL\Delta\mathrm{PPL} (%)TPS↑\mathrm{TPS}\uparrow (bsz=1)
Official FP828.7627.67+0.5089.3
RedHat AI INT418.13—+3.25105.0
Low-bit releases
sdkyuan QAT Q2_09.04—+22.05101.4
ISTA-DASLab IQ2_XS8.728.77+17.0555.0
Unsloth UD-IQ2_S8.688.57+13.8355.2
Unsloth UD-IQ2_XXS7.657.41+25.2656.6

Table A5. Released Qwen3.8 27B quantizations under the pilot protocol. ΔPPL\Delta\mathrm{PPL} is the percentage increase over BF16 (9.9227) on held-out C4. TPS is batch-one decode speed, 127/LL, with LL the 128-token first-to-last interval.

A QTIP pilot package occupies 6.96 GiB with PPL 20.84% above BF16, but retains BF16 weights during execution; agent pilots also retain high-precision copies. The low-bit GGUF releases keep resident storage within 0.24 GiB of package size. These measurements motivate separate allowances for saved packages and resident weights.

The released quantizations used in the main study are identified with their verification results in Table A3.

Additional Experiments and Decision Analysis

Repeated Agentic Design

We repeat GLM 5.3 and DeepSeek V4.1 Flash on one dense and one MoE task (Table A6). Four fresh trials use empty workspaces, the original checkpoints, prompts, and verification protocol, Claude Code through Fireworks, maximum reasoning effort, eight hours, and one H200 NVL.

AgentTrialPPLHE/TQ/GSMean
Qwen3.8 27B / 12 GiB
DeepSeekOrig.3.39974/56/8672.00
DeepSeekRepeat3.39268/56/8870.67
GLMOrig.3.38382/64/9680.67
GLMRepeat3.35670/56/8068.67
Qwen3.5 122B-A10B / 48 GiB
DeepSeekOrig.2.80382/76/9684.67
DeepSeekRepeat2.82086/72/9684.67
GLMOrig.2.81478/74/9682.67
GLMRepeat2.81982/74/9684.00

Table A6. Original and repeated eight-hour designs on one H200 NVL. Final verification measures perplexity (PPL) and accuracy (%). HE/TQ/GS denote HumanEval/TriviaQA/GSM8K, with fifty questions each; Mean averages their accuracies.

All four repeats pass final verification. Their PPL values lie within 0.81% of the corresponding original designs, with the largest shift in GLM’s dense-model repeat.

Controlled Quantization Ablations

We compare clipping-range selection and GPTQ error compensation in GLM’s repeated dense and MoE designs (Table A7). Paired variants share source weights, calibration, precision allocation, group sizes, and inference. The MoE scales and channel order remain fixed.

SettingPPLGSM8KHETQMean
Qwen3.8 27B / 12 GiB: three-bit MLP clipping
MSE search3.3556080705668.67
Min–max3.3693172605462.00
Qwen3.5 122B-A10B / 48 GiB: compensation
GPTQ2.8192496827484.00
No compensation2.8344496767883.33

Table A7. Quantization ablations of GLM’s repeated designs. PPL and accuracies (%) use final verification. HumanEval (HE), TriviaQA (TQ), and GSM8K each use fifty questions; Mean averages their accuracies. Paired variants have identical package/resident storage: 11.843/11.859 GiB (dense) and 47.835/47.854 GiB (MoE). The MoE control retains the selected scales while removing GPTQ error compensation.

Clipping-range refinement in the dense model and GPTQ error compensation in the MoE model both lower verification PPL at identical model storage. Compensation reduces negative log-likelihood on 197 of 256 MoE verification contexts.

Agent Responses to Feedback

Figure A2 summarizes actions after local rankings and reward feedback, together with subsequent generation and execution tests.

Figure A2. Agent responses during compression design. (a) Actions following 79 comparisons of local rankings with reward feedback in 26 of 36 runs; hatched bars denote ranking reversals, unhatched bars confirmations. Actions may co-occur. (b) Circles indicate attempted repairs; diamonds indicate matched local generation gains (eight cases, upper three rows) or execution within both storage limits (ten episodes, lower three rows).

Six of eight local generation regressions prompt revisions. Opus and Kimi improve on matched local tests while remaining below their references. Ten execution or storage episodes later complete within both limits, including one warmup retry.

Resident Storage and Peak Device Memory

For both releases, most additional device memory is allocated during loading, with smaller increases during prefill and decode (Table A8). Runtime allocations include caches, recurrent states, and temporary workspace.

ModelResident storageSampled device peak LoadSampled device peak PrefillSampled device peak Decode
Qwen3.810.95313.00013.34913.353
Qwen3.543.34945.33245.50645.510

Table A8. Resident storage and device peaks (GiB) for Qwen3.8 27B GSQ-RCO IQ3_S and Qwen3.5 122B-A10B Unsloth UD-IQ3_S. Values are medians of three cold starts. Device usage is sampled every 10 ms at batch one with 2,048 input and 128 output tokens and a 4,096-token single-sequence cache.

References

  1. [1]Saleh Ashkboos, Maximilian Croci, Marcelo Gennari de Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, pages 11682–11701, 2024a.arxiv.org/abs/2401.15024
  2. [2]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. Advances in Neural Information Processing Systems, 37: 100213–100240, 2024b.arxiv.org/abs/2404.00456
  3. [3]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE-bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations, pages 50466–50494, 2025.arxiv.org/abs/2410.07095
  4. [4]Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QuIP: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396–4429, 2023.
  5. [5]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  6. [6]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  7. [7]Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan, Michael Helcig, Eldar Kurtic, Maximilian Kleinegger, and Dan Alistarh. GSQ: Highly-accurate low-precision scalar quantization for LLMs via gumbel-softmax sampling. arXiv preprint arXiv:2604.18556, 2026.
  8. [8]Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051. Association for Computational Linguistics, 2023.
  9. [9]Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael Mahoney, and Kurt Keutzer. HAWQ-V2: Hessian aware trace-weighted quantization of neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 18518–18529, 2020. https://proceedings.neurips.cc/paper/2020/hash/d7c703536718b95308130ff2e5cf9ee-Abstract.html.
  10. [12]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.
  11. [13]Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments. https://doi.org/10.5281/zenodo.20953922, 2026. https://harborframework.com/.
  12. [15]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  13. [16]Mark Horowitz. 1.1 computing’s energy problem (and what we can do about it). In IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10–14, 2014.
  14. [17]Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. In International Conference on Machine Learning, pages 20271–20309. PMLR, 2024.
  15. [18]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kevin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, pages 54107–54157, 2024.DOI

Paper details

vlnc:W_5jaxmb9z
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code, Codex

View statement
Looking for related papers…
Contents