betaFeedback

WorksComputer Science

Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models

Abstract

Depth pruning removes entire transformer layers to make large language models faster, and recent methods recover quality by replacing the removed layers with a training-free linear map. Which layers to remove, however, is chosen with heuristic scores whose distance from the best choice is unknown. We evaluate every subset of layer groups exactly (about 53,000 pruned models across four families, with and without repair) and find that the best layers to remove depend on the repair: linear repair absorbs the interaction among nearby layers and turns the optimum from a split layout into a contiguous block, so methods that choose layers without the repair miss the repaired optimum by up to 189%. We therefore select layers under the repair that will be applied. A greedy search at the cost of SLEB lowers perplexity by 20% to 66% relative to ReplaceMe on five models up to 27B parameters, and with a task objective it is up to 25 points more accurate given the same data. A second-order expansion of the loss yields a score that selects the repaired block from one forward and one backward pass of the intact model.

Introduction

Depth pruning removes whole transformer layers. Unlike width pruning or sparsity, it reduces latency and memory on standard hardware without special kernels, because the pruned model is a shorter model of the same architecture [27, 36, 19]. Recent methods add a training-free repair: they replace the removed layers by a linear map fitted on calibration activations and fold the map into the weights of a kept layer, so the repaired model has the same parameter count as the pruned one [4] (Shopkhoev et al. 2025; Chen et al. 2026).

Every depth-pruning method must choose which layers to remove. For LL layers and nn removed layers there are (Ln)\binom{L}{n} choices, and each choice must be scored on a pruned model. Published methods therefore use proxies: the similarity of a layer’s input and output [27, 15] (Shopkhoev et al. 2025), greedy removal [36], evolutionary search [35], or game-theoretic estimates of each layer’s contribution [34, 10]. How far these choices are from the best one is unknown, and recent work concludes that more elaborate search adds little [20].

These proxies score each layer in the unrepaired model. With a linear repair, the interactions among the removed layers change: a map absorbs the joint effect of contiguous layers, so the layers that are best to remove together differ from those individually least important. The gap between the proxy and the true optimum therefore depends on the repair, and it is unknown because the true optimum has not been computed. We close this gap. At a granularity of two-layer groups the number of subsets is small enough to evaluate every one: about 53,000 pruned models over Qwen3-1.7B, Gemma-3-4B, Llama-3.1-8B and Qwen3-14B, each scored after plain removal and after the least-squares repair of ReplaceMe (Shopkhoev et al., 2025). The exact optima show that the answer depends on the repair (Figure 1): without repair, removing two nearby layers costs more than the sum of their individual costs, so the best subset spreads the removal over separate runs and a contiguous block is close to the worst choice. With repair, one map absorbs the interaction inside a run, and the best subset becomes a block, or a block and one short run. Published methods choose layers without the repair, and so miss the repaired optimum by up to 189%.

Line graph comparing the perplexity of the best contiguous block with the exact optimum, with and without repair

Figure 1. Repair changes which set of removed layers is best. Perplexity of the best contiguous block relative to the exact optimum over all subsets, at every enumerated budget of four models. Without repair (grey) a block is up to 6.9 times worse than the optimum. With ReplaceMe’s linear repair (red) the best block is the optimum at 9 of the 18 budgets and within 5% of it at 13.

The fix follows directly: evaluate every candidate subset on the repaired model that will be deployed (Figure 2). A greedy search that does so costs as much as SLEB and finds the exact repaired optimum, or comes within 0.2% of it, at every enumerated budget on three of four models. A second-order expansion of the repaired loss turns the same principle into a score computed from one forward and one backward pass of the intact model. Our contributions are:

Overview of exact optima, repair shifts, and layer selection under repair

Figure 2. Overview. (a) Evaluating every subset of layer groups shows critical layers and local super-additivity, so the optimum without repair splits the removal into separate runs. (b) ReplaceMe’s least-squares map replaces a run by a linear map folded into the down-projection of the kept layer before it; it absorbs the interaction inside the run, and the repaired optimum becomes one block. (c) RGS evaluates every candidate on its repaired model at the cost of SLEB; RFS ranks blocks by the Fisher-weighted residual of their repair, computed on the intact model only.

  • The structure of the exact optimum. A Möbius decomposition of the loss increase from removal shows two interactions in all four families: a few critical layers whose joint loss increase is far below the sum of their individual loss increases, and local super-additivity among all other layers, which decays with distance in depth with a length of 3 to 6 layers (§ 4). As a result, at 4 to 6 removed groups the best contiguous block is 18% to 588% worse than the optimum.

  • Linear repair makes the optimum contiguous. One least-squares map per run absorbs the interaction among the run’s layers, and the repaired optimum becomes one block, or one block and one short run (§ 5). Selection must therefore account for the repair: choosing layers without the repair and then repairing them is 5% to 33% worse than the repaired optimum, and ReplaceMe’s cosine criterion is 11% to 189% worse.

  • Selection methods. Greedy selection under repair (RGS), at the cost of SLEB, has 20% to 66% lower perplexity than ReplaceMe on four models and 54% to 56% lower on a 27B hybrid-attention model, and with a task objective is more accurate at every budget, by up to 25 points on the 27B model. A second-order expansion of the Kullback–Leibler divergence gives a score (RFS) that selects the repaired block from the gradients of the intact model (§ 6).

  • Evaluation beyond the selection data. We test the methods with ReplaceMe’s default calibration, on tasks that the objective does not see, on C4, after LoRA recovery, and on a 27B

model that was not used for any design choice, and we report the cases where a baseline is better (§ 7).

Related work

Choosing layers to remove. ShortGPT removes the layers whose output is most similar to their input [27]; Gromov et al. (2025) remove the contiguous block with the smallest angular distance between its input and output, and LLM-Streamline [3], LaCo [39] and CLP [26] remove or merge contiguous blocks by design. SLEB removes layers greedily by the perplexity of the pruned model [36]; BlockPruner does the same for attention and MLP sublayers [42]. EvoPress searches over subsets and observes that removal is not additive [35], and Shapley-value estimates model the interaction with sampling or surrogates [34, 10]. Kim et al. (2026) call the subset problem NP-hard and find that search gains little over one-shot scores without repair. We compute the optimum exactly, decompose its interactions, and show that the gain of search depends on the repair.

Training-free repair. ReplaceMe fits one linear map per removed block by least squares and folds it into the down-projection of the last kept layer (Shopkhoev et al., 2025). LinearPatch [4] and Prune&Comp (Chen et al., 2026) correct scale and outliers at the pruning interface; Prune&Comp alternates pruning and compensation, so each candidate is scored on a model that earlier steps have compensated, with one scalar per layer. Other repairs use low-rank or shifted maps [24, 40, 1], and SubFit selects submodules by the residual of their fitted replacement [9]. These methods repair after the layers are chosen, or choose with a criterion that does not evaluate the repaired model. ReplaceMe reports that its cosine criterion finds optimal or near-optimal 8-layer blocks on Llama 3 8B (Shopkhoev et al., 2025, App. A.10); over all block lengths and four families we find it 11% to 189% worse in perplexity than the repaired optimum.

Second-order saliency and model reduction. Optimal Brain Damage and Optimal Brain Surgeon score the removal of a weight by a second-order expansion of the loss [21, 16], and OBC, GPTQ and SparseGPT apply it layer by layer [11, 13, 12]. Fisher pruning and Taylor importance score the removal of an activation aa by ∑(g⋅a)2\sum(g \cdot a)^2 [38, 29], and BRECQ weights reconstruction error by the Fisher information [22]. Our score applies this expansion to the error that remains after repair. Balanced truncation reduces a linear system by weighting the error of the reduced model by how much the output observes it [30]; our score has the same form, with the loss gradient in the role of the observability.

Problem setup

Pruning and repair. A decoder-only model with LL layers updates a residual stream hℓ∈Rdh_\ell\in\mathbb{R}^d at each token by hℓ=hℓ−1+fℓ(hℓ−1)h_\ell= h_{\ell-1} + f_\ell(h_{\ell-1}). We partition the layers into GG contiguous groups of one or two layers. Removing a set SS of groups skips their layers; a run is a maximal set of adjacent removed groups. Linear repair (Shopkhoev et al., 2025) replaces a run of layers a..ba..b by a map T∈Rd×dT \in\mathbb{R}^{d \times d} on the MLP output ma−1m_{a-1} of the last kept layer before the run, which then outputs y+ma−1Ty + m_{a-1}T instead of y+ma−1y + m_{a-1}, with y=ha−1−ma−1y = h_{a-1} - m_{a-1}. The map solves

Ta,b=arg min⁡T ∥Ma−1T−(Hb−Ya−1)∥F2+λ∥T∥F2,(1)T_{a,b} = \underset{T}{\operatorname{arg\,min}}\ \left\|M_{a-1}T-(H_b-Y_{a-1})\right\|_F^2+\lambda\|T\|_F^2, \tag*{(1)}

where the rows of Ma−1M_{a-1}, Ya−1Y_{a-1} and HbH_b are activations of the intact model at calibration tokens. TT folds into the down-projection of layer a−1a-1, so the repaired model has the architecture of the pruned one. Each run has its own map; a run that starts at the first layer cannot be repaired.

Objective and data. The loss increase of a subset is the mean token loss of the pruned model minus that of the intact model on 16 WikiText-2 calibration sequences of 256 tokens [28]. Every method selects layers on these sequences, and we report perplexity (PPL) on 16 other sequences of the WikiText-2 test split. Maps are fitted on 128 WikiText-2 training sequences with λ=10−3\lambda= 10^{-3} times the mean diagonal of MTMM^\mathsf{T}M. Accuracy is the mean over ARC-Easy, ARC-Challenge, HellaSwag, PIQA and WinoGrande [8, 41, 2, 33], 1,000 items each, with the prompts of lm-evaluation-harness (Sutawika et al., [37]); one standard error of the mean is about 0.7 points. BoolQ (Clark et al., [7]) is reported separately, because pruning shifts the yes/no prior and moves it by 20 points or more independently of the other tasks.

Exact enumeration. We use Qwen3-1.7B and Qwen3-14B (Qwen Team, 2025), Gemma-3-4B (Gemma Team, 2025) and Llama-3.1-8B-Instruct (Llama Team, [25]) with two-layer groups (14, 17, 16 and 20 groups), and evaluate every subset of up to 6 removed groups (4 on Qwen3-14B), without and with repair (Table 14). We also enumerate Qwen3-1.7B at single-layer granularity (28 groups) up to 3 removed layers; the same interaction structure appears at both granularities. We store the loss of every subset on every sequence, so any search method can be evaluated against the exact optimum without further model runs.

ModelLayersGroupsSubsets per budget
Qwen3-1.7B2814all, up to 6 removed
Qwen3-1.7B2828all, up to 3 removed
Gemma-3-4B3417all, up to 6 removed
Llama-3.1-8B3216all, up to 6 removed
Qwen3-14B4020all, up to 4 removed

Table 14. Exactly enumerated settings: every subset of groups up to the stated number of removed groups is evaluated on every calibration and test sequence, without and with repair; about 53,000 subsets in total.

The structure of the exact optimum

Interaction decomposition. Let c(S)c(S) be the loss increase from removing SS, with c(∅)=0c(\varnothing)=0. Its Möbius transform (Grabisch and Roubens, 1999)

μ(S)=∑R⊆S(−1)∣S∣−∣R∣c(R),c(S)=∑R⊆Sμ(R)(2)\mu(S)=\sum_{R\subseteq S}(-1)^{|S|-|R|}c(R),\qquad c(S)=\sum_{R\subseteq S}\mu(R) \tag*{(2)}

splits the loss increase into an individual term μ({i})\mu(\{i\}) for each group and an interaction term for each larger set; removal is additive exactly when every interaction term is zero. The interaction share of a budget, the mean absolute difference between c(S)c(S) and ∑i∈Sc({i})\sum_{i\in S}c(\{i\}) as a share of the mean absolute loss increase, rises from 21% to 50% at 2 to 6 removed groups on Qwen3-1.7B, 15% to 31% on Gemma, 7% to 25% on Llama and 11% to 22% (2 to 4 removed) on Qwen3-14B.

Critical layers. We call a group critical when its removal alone more than doubles perplexity: 4 of 14 groups in Qwen3-1.7B, 5 of 17 in Gemma, 3 of 16 in Llama and 3 of 20 in Qwen3-14B, usually the first ones. Their pairwise terms are large and negative, because the loss saturates once the model fails, and every exact optimum keeps them. We call the other groups non-critical.

Local super-additivity. Among non-critical groups the pairwise term μ({i,j})\mu(\{i,j\}) is positive and decays with the distance ∣i−j∣|i-j| in depth (Figure 3), with fitted decay lengths of 2.9 layers (Gemma)

Mean pairwise interaction term of two removed non-critical groups against their distance in depth

Figure 3. Mean pairwise interaction term of two removed non-critical groups against their distance in depth, with exponential fits over the first four distances (dashed). Removing two nearby groups costs more than the sum of their individual costs.

to 6.4 layers (Llama) and terms close to zero beyond about 10 layers. Pairwise terms explain 52% to 76% of the non-additivity at 3 removed groups; higher-order terms among nearby groups grow at heavier budgets.

A contiguous block is close to the worst choice. A block contains the most pairs within the interaction range, so under local super-additivity it has the largest interaction cost. At small budgets the optimum can be a block; from 4 removed groups on, every exact optimum in every family splits the removal into 2 or 3 runs, and the best block is 18% to 588% worse in perplexity (Table 1). Methods that remove one block by design (Gromov et al., [15]; Chen et al., [3]; Lu et al., [26]) therefore cannot find the optimum when the layers are not repaired.

Model3456
Qwen3-1.7B0%28%77%588%
Gemma-3-4B64%252%512%447%
Llama-3.1-8B32%34%76%190%
Qwen3-14B0%18%––

Table 1. Excess perplexity of the best contiguous block over the exact optimum without repair.

Implications for existing methods. One-shot scoring removes the groups with the smallest individual loss increase. These cluster in depth, so one-shot scoring removes an almost contiguous set: 33% above the optimum on Qwen3-1.7B and 581% on Gemma at 4 removed groups. A pairwise model assigns large negative terms to critical-layer pairs; its optimum is 189 to 226,000 times worse. SLEB evaluates actual pruned models and accounts for the interaction, but cannot undo an earlier removal: it finds the optimum in 13 of 20 budgets (Section D). The interaction remains when a removed layer is replaced by its mean update; it therefore comes from the lost computation (Section C).

Linear repair makes the optimum contiguous

Table 2 compares the exact optimum without repair, that optimum after repair, ReplaceMe’s choice, and the exact optimum under repair over every subset of non-critical groups.

ModelnnRemoval opt. as isRemoval opt. repairedReplaceMeRepaired opt.
Qwen3-1.7B (26.5)473.450.375.141.7
Qwen3-1.7B (26.5)5128.759.290.654.3
Qwen3-1.7B (26.5)6229.280.299.576.0
Gemma-3-4B (32.0)446.750.644.738.3
Gemma-3-4B (32.0)565.951.672.242.8
Gemma-3-4B (32.0)693.564.1149.851.9
Llama-3.1-8B (13.7)433.626.840.422.2
Llama-3.1-8B (13.7)553.032.561.928.3
Llama-3.1-8B (13.7)684.244.194.035.1
Qwen3-14B (15.0)423.125.621.319.1

Table 2. Perplexity at the heaviest enumerated budgets (nn removed groups; full model in parentheses). “Removal opt., repaired” repairs the best subset for plain removal. The last column is the exact optimum under repair.

Repair absorbs the interaction inside a run. One map is fitted to the input and output of a whole run, so the super-additivity among the run’s layers is part of what it fits. At the heaviest budget the interaction share falls from 62% to 36% on Qwen3-1.7B, 66% to 52% on Gemma, 57% to 28% on Llama and 25% to 15% on Qwen3-14B. Separately repaired runs still interact: the sum of their repaired loss increases explains the loss increase of a subset with R2R^2 of 0.8 to 0.9 at 2 removed groups but only −0.09-0.09 to 0.44 at 6, because each map assumes that the rest of the network is intact.

The repaired optimum is contiguous. Without repair, every optimum at 4 or more removed groups splits into 2 or 3 runs. Under repair it is one block (Qwen3-1.7B at 4 and 5 removed groups, Qwen3-14B at 4) or one block and one more run (Figure 8 in Section E). A contiguous block, which is close to the worst choice without repair, is close to the best choice with repair.

Figure 8. Groups removed by the exact optimum at the heaviest enumerated budget, without repair (grey) and with repair (red); light cells are critical groups. Repair turns two or three separate runs into one block, or a block and a short run, at 17% to 67% lower perplexity.

Selection must account for the repair. Two simpler approaches are worse. Repairing the best subset for plain removal is 5% to 33% worse than the repaired optimum, because that subset is split into runs that repair cannot absorb. Choosing the block with the smallest cosine distance between its input and output, as ReplaceMe does, is 11% to 189% worse. On Llama at 8 of 32 layers, a setting that ReplaceMe evaluates, its block is among the most accurate subsets under the WikiText objective (Table 3), in agreement with Shopkhoev et al. (2025, App. A.10), but its perplexity is 79% higher than that of the subset chosen under repair, and with a task objective the subset chosen under repair is 4.0 points more accurate. A selection method must therefore evaluate each subset on the repaired model that it deploys.

Qwen3-1.7B 7/28 PPL↓Qwen3-1.7B 7/28 Avg↑Qwen3-1.7B 10/28 PPL↓Qwen3-1.7B 10/28 Avg↑Gemma-3-4B 8/34 PPL↓Gemma-3-4B 8/34 Avg↑Gemma-3-4B 13/34 PPL↓Gemma-3-4B 13/34 Avg↑Llama-3.1-8B 8/32 PPL↓Llama-3.1-8B 8/32 Avg↑Llama-3.1-8B 12/32 PPL↓Llama-3.1-8B 12/32 Avg↑Qwen3-14B 10/40 PPL↓Qwen3-14B 10/40 Avg↑Qwen3-14B 15/40 PPL↓Qwen3-14B 15/40 Avg↑Qwen3.8-27B 16/64 PPL↓Qwen3.8-27B 16/64 Avg↑Qwen3.8-27B 24/64 PPL↓Qwen3.8-27B 24/64 Avg↑
Full model26.560.026.560.032.069.832.069.813.772.413.772.415.073.015.073.010.172.810.172.8
Same calibration text for every method (WikiText-2)
ShortGPT + R77.446.1130.040.444.750.887.242.739.459.594.046.848.249.2266.437.734.840.642.438.6
LaCo + R78.739.9571.336.257.054.0445.439.646.758.582.645.690.941.11119.136.4n/an/an/an/a
ReplaceMe53.745.668.740.944.750.8150.741.540.457.294.046.841.951.443.344.528.742.236.942.2
Prune&Comp + R63.546.7120.241.244.750.8192.340.642.257.087.045.645.451.476.642.228.148.933.942.9
SLEB + R40.345.959.141.850.554.769.144.723.656.237.848.427.249.143.143.013.265.717.157.4
RFS (ours)38.744.954.341.138.248.197.640.624.351.295.939.421.953.034.045.813.466.415.755.5
RGS (ours)37.744.254.341.135.950.651.144.722.655.434.449.021.954.434.145.113.364.916.057.9
RGS vs. ReplaceMe−30%−1.5−21%+0.2−20%−0.2−66%+3.2−44%−1.8−63%+2.2−48%+3.0−21%+0.6−54%+22.7−56%+15.7
Selection on task training data†
ReplaceMe, task data51.943.868.740.971.345.8192.340.640.958.994.046.841.951.4251.338.828.742.236.942.2
RGS-T (ours)64.547.1110.041.768.159.9143.751.234.661.254.051.937.959.567.747.927.666.833.359.8
RGS-T vs. ReplaceMe, task data+24%+3.3+60%+0.9−5%+14.1−25%+10.6−15%+2.3−43%+5.1−10%+8.1−73%+9.1−4%+24.6−10%+17.6

Table 3. Main results: WikiText-2 perplexity (PPL) and the average zero-shot accuracy (%) over ARC-Easy, ARC-Challenge, HellaSwag, PIQA and WinoGrande (Avg; one standard error is about 0.7 points). Every method is repaired with the same least-squares maps (+R); the methods differ only in which layers they remove. The upper block selects on WikiText-2 calibration text; the lower block (†) selects on the training items of the five tasks, which are disjoint from the evaluation items. Within each block the best result is highlighted and the second best underlined; italic rows give the change relative to ReplaceMe. LaCo’s merge needs identical layers and does not apply to the hybrid 27B model. Per-task scores are in Table 13.

Selecting layers under repair

Greedy selection under repair

We therefore include the repair in the search. Greedy selection under repair (RGS) removes one layer at a time, as SLEB does, but evaluates each candidate subset after fitting and applying its repair maps (Algorithm 1). Maps are fitted once per run from cached activations of the intact model, so a candidate costs one forward pass over the calibration text and RGS costs as many evaluations as SLEB, ∑k=0n−1(L−k)\sum_{k=0}^{n-1}(L-k) for nn removed layers. On the exact tables RGS finds the repaired optimum at every budget on Qwen3-1.7B and Qwen3-14B, and is within 0.2% of it on Llama and within 3.4% on Gemma. The objective can be any loss on the repaired model. With a task objective, the cross-entropy of the correct choice among the choices on 200 training items per task (RGS-T), it selects for accuracy.

A second-order score from the intact model

Under repair the optimum is close to one block, so ranking blocks is sufficient, and a second-order expansion of the loss ranks them with the activations and gradients of the intact model. Let z∈RN×dz \in\mathbb{R}^{N \times d} be the residual stream after block a..ba..b at the NN calibration tokens of the intact model. The repaired model produces z+δz+\delta there, with δt=yt+mtTa,b−zt\delta_t = y_t + m_t T_{a,b} - z_t the residual of the repair at token tt; the layers after the block are unchanged.

Proposition 1. Let ps(z)p_s(z) be the next-token distribution at position ss as a function of the residual stream, and D(δ)=∑sKL⁡(ps(z)∥ps(z+δ))D(\delta) = \sum_s \operatorname{KL}(p_s(z) \mathbin\Vert p_s(z + \delta)). Then D(0)=0D(0) = 0, ∇D(0)=0\nabla D(0) = 0 and

D(δ)=12vec⁡(δ)TFvec⁡(δ)+O(∥δ∥3),(3)D(\delta) = \frac{1}{2}\operatorname{vec}(\delta)^\mathsf{T}F\operatorname{vec}(\delta) + O(\lVert\delta\rVert^3), \tag*{(3)}

where F=∑sEy∼ps[∇zlog⁡ps(y)∇zlog⁡ps(y)T]F = \sum_s \mathbb{E}_{y\sim p_s}[\nabla_z \log p_s(y)\nabla_z \log p_s(y)^\mathsf{T}] is the Fisher information of the downstream network with respect to the residual stream.

The proof is in Section A. We replace the expectation over yy by the observed tokens (the empirical Fisher) and keep one rank-one block per position, F≈blockdiag⁡t(gtgtT)F \approx\operatorname{blockdiag}_t(g_tg_t^\mathsf{T}) with gt=∇zt∑sℓsg_t = \nabla_{z_t}\sum_s \ell_s the gradient of the summed token loss, so that each position plays the role of a sample as in Fisher pruning (Theis et al., [38]) and Taylor importance (Molchanov et al., [29]). The result is the repair Fisher score (RFS) of a block:

s(a,b)=∑t=1N(gt⋅δt)2.(4)s(a,b) = \sum_{t=1}^{N}(g_t\cdot\delta_t)^2. \tag*{(4)}

The least-squares map of Equation (1) minimises the Euclidean norm of δ\delta; Equation (4) measures the same residual in the metric of the downstream Fisher information, which weights each direction of the residual stream by how much the loss responds to it. This is the weighting that balanced truncation applies to the error of a reduced system, with the observability Gramian in the role of FF (Moore, [30]). The unweighted residual and the first-order term ∑tgt⋅δt\sum_t g_t\cdot\delta_t give inconsistent block rankings across models (§ 7.3).

Experiments

Experimental setup

Models and budgets. We use the four enumerated families of § 3 and Qwen3.8-27B, which has 64 layers with three linear-attention layers to each full-attention layer and was not used for any design choice. We remove 25% and 37.5% of the layers at single-layer granularity.

Baselines. ShortGPT’s block influence (Men et al., [27]), LaCo (Yang et al., [39]), ReplaceMe’s cosine block (Shopkhoev et al., 2025), Prune&Comp (Chen et al., 2026) and SLEB (Song et al., [36]). Every method is repaired with the same least-squares maps (+R), so the methods differ only in which layers they choose. For LaCo we remove the layers that its merge criterion selects, because this is better than its parameter merge at every budget we test. For the task objective we also give ReplaceMe’s criterion the same task training texts.

Metrics and implementation. We report WikiText-2 perplexity on the test sequences, C4 perplexity, and the average accuracy over five zero-shot tasks (§ 3). All models run in bfloat16 on one 48 GB GPU, except Qwen3.8-27B, which runs on two (Section G).

Main results

Perplexity. On the four enumerated families RGS has the lowest perplexity at 7 of 8 settings, with RFS 0.3% closer at the eighth; it is 20% to 66% below ReplaceMe and 4% to 29% below SLEB with repair (Table 3; Figure 9 in Section E). SLEB uses the same greedy search without repair, so this difference comes from the repair in the search. It is 20% to 73% below Prune&Comp and 37% to 97% below LaCo, both with repair, on every model they apply to. The gap to ReplaceMe is largest where the repaired optimum is not one block: on Gemma at 13 of 34 layers RGS removes a block of 8 layers and 5 separate ones and has one third of ReplaceMe’s perplexity. On Qwen3.8-27B it is 54% and 56% below ReplaceMe, whose criterion picks the first layers on this architecture; RGS matches SLEB with repair at 16 layers (within 0.5%) and is 7% better at 24. Across all five models and both budgets, including the repair in the search consistently lowers perplexity.

Perplexity relative to ReplaceMe for five model families and four methods

Figure 9. Perplexity relative to ReplaceMe (log scale), every method repaired by the same maps. RGS is 20% to 66% lower on the four enumerated families and 54% to 56% lower on Qwen3.8-27B.

Accuracy. Given the same task training data, RGS-T is more accurate than ReplaceMe’s criterion at all 8 settings, by 0.9 to 14.1 points (lower block of Table 3, Figure 4), and also has lower perplexity at 6 of them. Both methods use the same data, so the gain comes from selection under repair. RGS-T has the highest average of all methods at 7 of 8 settings and beats plain ReplaceMe at all 8, by 0.9 to 9.8 points; on Gemma at 8 of 34 layers it keeps 86% of the full model’s accuracy against ReplaceMe’s 73%.

Average accuracy over the five zero-shot tasks

Figure 4. Average accuracy over the five zero-shot tasks. Labels give the gain of RGS-T over ReplaceMe’s criterion given the same task data (0.9 to 14.1 points); dashed lines mark the full model.

The accuracy gain transfers to the 27B model and to tasks outside the objective. On Qwen3.8-27B, RGS-T keeps 92% of the full model’s accuracy at 16 of 64 layers and 82% at 24, 24.6 and 17.6 points above ReplaceMe given the same data, and above every other method at both budgets. WikiText perplexity and task accuracy favour different layers (Table 13). Built from ARC-Easy, HellaSwag and PIQA only, the task objective still beats ReplaceMe on ARC-Challenge and WinoGrande at both budgets on three of four models (Table 9). Selection under repair thus improves both perplexity and accuracy, and the gain generalises across models, budgets and tasks.

ModelnnReplaceMeRGS-T, 3 tasksRGS-T, 5 tasks
Qwen3-1.7B738.140.040.0
1037.037.237.1
Gemma-3-4B841.052.152.6
1337.343.243.6
Llama-3.1-8B853.454.354.3
1247.947.347.3
Qwen3-14B1042.150.052.2
1538.943.542.2

Table 9. Mean accuracy (%) on ARC-Challenge and WinoGrande when the task objective uses only ARC-Easy, HellaSwag and PIQA (3 tasks); one standard error is about 1.1 points. The highlight compares ReplaceMe with the 3-task objective.

Method25% of layers removed37.5% of layers removed
PPL↓ARC-eARC-cHSPIQAWGAvg↑PPL↓ARC-eARC-cHSPIQAWGAvg↑
Qwen3-1.7B, 28 layers: 7 and 10 removed
Full model26.570.042.653.972.860.760.026.570.042.653.972.860.760.0
ShortGPT + R77.446.629.838.461.054.646.1130.035.824.434.656.051.240.4
ReplaceMe53.747.125.542.462.650.645.668.734.422.438.257.651.740.9
SLEB + R40.342.826.043.564.352.745.959.136.222.040.058.252.541.8
RFS (ours)38.742.424.342.762.053.044.954.335.821.438.656.753.041.1
RGS (ours)37.741.624.141.861.751.744.254.335.821.438.656.753.041.1
RGS-T (ours)64.549.328.042.763.552.047.1110.039.923.637.656.850.741.7
Gemma-3-4B, 34 layers: 8 and 13 removed
Full model32.078.957.165.278.169.669.832.078.957.165.278.169.669.8
ShortGPT + R44.753.624.652.266.357.550.887.240.222.941.858.150.442.7
ReplaceMe44.753.624.652.266.357.550.8150.736.823.340.455.651.341.5
SLEB + R50.561.432.152.870.656.754.769.142.325.442.861.551.744.7
RFS (ours)38.246.824.149.066.953.648.197.634.920.538.456.153.140.6
RGS (ours)35.952.327.151.869.053.050.651.142.722.544.361.552.344.7
RGS-T (ours)68.169.544.255.169.961.059.9143.759.434.144.465.253.151.2
Llama-3.1-8B-Instruct, 32 layers: 8 and 12 removed
Full model13.779.755.671.082.573.272.413.779.755.671.082.573.272.4
ShortGPT + R39.459.340.356.570.471.159.594.038.231.642.058.164.146.8
ReplaceMe40.456.337.254.468.269.757.294.038.231.642.058.164.146.8
SLEB + R23.661.334.058.772.654.556.237.846.626.348.566.054.648.4
RFS (ours)24.352.427.854.369.451.951.295.932.723.737.154.049.739.4
RGS (ours)22.659.931.357.773.554.455.434.448.425.648.767.155.249.0
RGS-T (ours)34.668.141.156.173.167.661.254.052.332.047.565.362.651.9
Qwen3-14B, 40 layers: 10 and 15 removed
Full model15.083.260.867.980.373.073.015.083.260.867.980.373.073.0
ShortGPT + R48.246.832.046.060.261.149.2266.430.123.630.550.753.737.7
ReplaceMe41.955.030.651.366.653.751.443.342.526.643.159.151.244.5
SLEB + R27.248.628.149.765.753.649.143.137.924.243.857.451.743.0
RFS (ours)21.954.832.152.471.454.253.034.046.324.844.662.051.445.8
RGS (ours)21.962.931.554.171.052.554.434.143.223.946.860.251.645.1
RGS-T (ours)37.968.544.453.471.260.159.567.749.230.043.562.454.547.9
Qwen3.8-27B, hybrid attention, 64 layers: 16 and 24 removed
Full model10.174.458.574.181.975.372.810.174.458.574.181.975.372.8
ShortGPT + R34.833.623.141.353.351.540.642.432.321.838.152.448.538.6
ReplaceMe28.733.822.042.558.354.442.236.936.424.542.555.751.942.2
SLEB + R13.274.849.662.677.963.865.717.165.837.756.171.356.057.4
RFS (ours)13.477.448.463.377.465.366.415.758.134.155.371.258.855.5
RGS (ours)13.373.849.763.077.360.664.916.066.937.255.674.255.557.9

Table 13. Per-task results behind Table 3: WikiText-2 perplexity (PPL) and zero-shot accuracy (%) on ARC-Easy, ARC-Challenge, HellaSwag (HS), PIQA and WinoGrande (WG), 1,000 items each. Within each model, the best pruned result is highlighted and the second best underlined.

MMLU. The selection objective decides which layers are kept. 5-shot MMLU (Hendrycks et al. [17]) depends on middle layers that have the smallest effect on WikiText perplexity: on Llama and Qwen3-14B, the WikiText subsets of RGS and SLEB remove them and score 24% to 28%, and the deep blocks of ShortGPT keep 67.1% and 68.4% at 25% removed, as in Gromov et al. [15] (Table 12). With an MMLU objective on validation questions, RGS removes deep layers instead and is the most accurate method at 7 of 8 settings, with 53.3% against at most 26.7% on Qwen3-14B at 15 layers (Section E). The selection objective therefore controls which capabilities are preserved after pruning.

Model (full)RemovedReplaceMeShortGPT + RSLEB + RWikiTextTasksMMLU
Qwen3-1.7B (63.5)724.036.124.725.126.731.3
1024.123.824.824.023.825.9
Gemma-3-4B (60.6)824.124.127.525.436.353.0
1323.623.823.725.624.727.2
Llama-3.1-8B (68.4)866.367.128.427.048.467.6
1258.458.423.524.031.662.3
Qwen3-14B (80.8)1023.468.425.924.737.575.1
1525.124.626.024.026.753.3
Qwen3.8-27B (83.5)1624.1–54.852.457.6–
2424.7–30.226.031.2–

Table 12. 5-shot MMLU accuracy (%) on 50 test questions per subject (2,850 questions) of the subsets in Table 3, and of RGS with an MMLU objective: the crossentropy of the correct letter on 5 validation questions per subject; the other methods select without MMLU data. Chance is 25%. All repairs are fitted on WikiText-2. –: not run.

The repair Fisher score

Against the exact repaired loss increase of every block of non-critical groups, the lowest RFS block is within 5% of the best at 19 of 22 block lengths; ReplaceMe’s cosine distance is within 5% at 2. The Spearman correlation is 0.53 to 0.86 for the score and −0.21 to 0.36 for the cosine distance (Figure 5). In Table 3 the score’s block, chosen without evaluating a pruned model, is within 3% of RGS at 4 of 8 settings and beats ReplaceMe at

Ranking blocks under repair without search

Figure 5. Ranking blocks under repair without search. (a) Spearman correlation of each score with the exact repaired loss increase of every block of a given length, averaged over lengths (error bars: one standard deviation over lengths). (b) Share of block lengths at which the lowest-scoring block is within 5% of the best block.

7 of 8. At the largest budgets (Gemma at 13 of 34, Llama at 12 of 32), RGS adds layers outside the block and reaches one half to one third of the score’s perplexity, so at heavy budgets the search justifies its cost. The Fisher score thus provides a strong default when a single block is sufficient, and the greedy search extends the method to heavier budgets.

ReplaceMe’s default calibration

ReplaceMe by default fits its maps on SlimOrca conversations (Lian et al. [23]) without a ridge. With this fit, RGS has 16% to 73% lower perplexity on Llama and Gemma (Table 6). With a SlimOrca objective, RGS differs from ReplaceMe only in

ModelnnReplaceMeRGS
Gemma-3-4B855.9 / 52.447.0 / 53.6
13243.7 / 41.670.5 / 47.7
Llama-3.1-8B863.7 / 60.226.1 / 56.8
12163.1 / 48.844.4 / 51.0

Table 6. ReplaceMe’s calibration (SlimOrca conversations, unregularised least squares) for every method. Perplexity / average accuracy (%).

which layers it removes; its perplexity is lower at 7 of 8 settings and its accuracy at 6 of 8 (Table 7). The advantage of selection under repair therefore holds across calibration data and fitting procedures.

ModelnnReplaceMeRGS, SlimOrca
Qwen3-1.7B750.5 / 45.350.9 / 47.2
10124.9 / 42.197.8 / 41.6
Gemma-3-4B855.9 / 52.454.7 / 54.8
13243.7 / 41.693.6 / 47.5
Llama-3.1-8B863.7 / 60.228.3 / 57.7
12163.1 / 48.849.3 / 50.7
Qwen3-14B1074.2 / 48.233.5 / 57.3
1587.4 / 44.254.7 / 45.7

Table 7. ReplaceMe’s calibration with a SlimOrca objective for RGS, disjoint from the fit data; selection and fit use conversation text only. Perplexity / average accuracy (%).

Recovery fine-tuning and a second corpus

RGS has lower C4 perplexity (Raffel et al., [32]) than ReplaceMe at both budgets on all four models, even though it selects on WikiText. After 200 LoRA steps (Hu et al., [18]) on C4 with three seeds, RGS keeps lower mean perplexity at 7 of 8 settings (Figure 6), by 16% to 30% on Llama, 2.5% to 19% on Gemma and 1% to 2% on Qwen3-1.7B. The benefit of better layer selection therefore persists after recovery fine-tuning.

C4 perplexity after repair and after recovery fine-tuning

Figure 6. C4 perplexity (a) after repair and (b) after 200 LoRA steps on C4 (mean of three seeds; error bars, one standard deviation). Labels give the change of RGS against ReplaceMe: recovery shrinks the gap, and RGS stays lower at 7 of 8 settings.

Ablations

Table 4 removes one component of RGS at a time on the 8 settings. The repair inside the search matters most: SLEB, which searches without it, has 19% higher perplexity on average (up to 41%). One-shot scoring under repair is 21% worse and ReplaceMe’s criterion 73% worse. Removing the layers that RGS selects without the repair raises perplexity by a factor of four, so the selected subset depends on the repair.

VariantPPL / RGSWorstAvgEvals
RGS1.001.0048.1303
+ shift refinement0.9951.0048.4303
− repair in the search (SLEB + R)1.191.4148.0303
− search (one-shot + R)1.211.7346.534
− search, block criterion (ReplaceMe)1.732.9547.30
− repair at deployment3.9814.9–303

Table 4. Ablation of RGS on the 8 settings of the four enumerated families. PPL / RGS: geometric mean of the perplexity ratio to RGS; Worst: the largest ratio; Avg: mean five-task accuracy (%); Evals: mean number of pruned models evaluated during selection.

Cost

The maps fold into existing weights, so at a given budget every method in Table 3 gives a model of the same size and speed. Removing 25% of the layers gives 1.3× speedup and removing 37.5% gives 1.55×; at 24 of 64 layers, Qwen3.8-27B shrinks from 53.8 to 35.6 GB and fits on one 48 GB GPU (Table 11). RGS evaluates as many pruned models as SLEB and RFS needs one forward and one backward pass (Section E).

ModelRemovedWeights (GB)KV (KB/tok.)SpeedupPPL
PrefillDec. 1Dec. 16
Qwen3-1.7B7/283.4→2.73.4 \to2.7112→84112 \to841.321.341.3237.72
10/283.4→2.43.4 \to2.4112→72112 \to721.551.531.5454.25
Llama-3.1-8B8/3216.1→12.616.1 \to12.6128→96128 \to961.321.301.3122.65
12/3216.1→10.816.1 \to10.8128→80128 \to801.581.541.5534.43
Qwen3-14B10/4029.5→22.929.5 \to22.9160→120160 \to1201.301.311.3121.96
15/4029.5→19.629.5 \to19.6160→100160 \to1001.571.551.5634.26
Qwen3.8-27B16/6453.8→41.753.8 \to41.764→3664 \to361.291.311.2913.26
24/6453.8→35.653.8 \to35.664→3264 \to321.551.581.5516.03

Table 11. Inference cost of the folded RGS models in bfloat16 on RTX A6000 GPUs (two for Qwen3.8-27B): weights, KV cache per token, and speedup over the full model for prefill of 2,048 tokens and for decoding 256 tokens at batch 1 and 16. PPL is the WikiText-2 perplexity of the folded model; it matches the model with the maps applied as separate operations to within 0.05. Gemma-3 normalises the MLP output before the residual addition, so its maps cannot be folded and it is not listed. Only the full-attention layers of Qwen3.8-27B keep a KV cache.

Conclusion

The exact optimum of depth pruning depends on the repair: local super-additivity makes a contiguous block close to the worst choice without repair and close to the best with it. Evaluating each choice on the repaired model that will be deployed lowers perplexity by 20% to 66% relative to ReplaceMe at the cost of SLEB, and a second-order expansion of the repaired loss gives a score computed from the intact model.

The core finding applies whenever the repair alters the cost landscape of the removed layers. Future work can test whether it holds for low-rank or shifted repairs. The repair Fisher score shows that repair quality can be measured from the intact model alone, which opens a path to joint optimisation of layer selection and the repair objective.

Limitations

Exact optima are available only at two-layer granularity up to 6 removed groups and at single-layer granularity up to 3; at heavier budgets we compare methods with each other. The structural results concern perplexity: at two-layer granularity the perplexity optimum is the most accurate subset, or within one standard error of it, in 10 of 14 cases, and in the other four a different subset is 2 to 7 points more accurate. We test one repair, ReplaceMe’s least-squares map; two schemes that refit later maps on the already repaired network did not improve on it, and newer repairs ([4, 40] (Chen et al. 2026)) are not tested. We do not compare with BlockPruner or SubFit. Recovery used one length, 200 LoRA steps. ReplaceMe’s example configuration fits on conversations of up to 1,024 tokens with the model in 4 bits; we fit on blocks of 256 tokens with the model in bfloat16. We evaluate only English text and multiple-choice tasks; multilingual text and open-ended generation remain untested.

References

Proof of Proposition 1

Fix a position ss and write p=ps(z)p = p_s(z) and q(δ)=ps(z+δ)q(\delta) = p_s(z + \delta). Then

KL⁡(p∥q(δ))=∑ypylog⁡py−∑ypylog⁡qy(δ).\operatorname{KL}(p \mathbin\Vert q(\delta)) = \sum_y p_y \log p_y - \sum_y p_y \log q_y(\delta).

At δ=0\delta= 0 the divergence is zero. Its gradient is −∑ypy∇log⁡qy(0)=−∑y∇qy(0)=−∇∑yqy(0)=0-\sum_y p_y \nabla\log q_y(0) = -\sum_y \nabla q_y(0) = -\nabla\sum_y q_y(0) = 0, because q(0)=pq(0) = p and the probabilities sum to one for every δ\delta. With ∇2log⁡qy=∇2qy/qy−∇log⁡qy∇log⁡qyT\nabla^2 \log q_y = \nabla^2q_y/q_y - \nabla\log q_y \nabla\log q_y^\mathsf{T} and ∑y∇2qy=0\sum_y \nabla^2q_y = 0, its Hessian is

−∑ypy∇2log⁡qy(0)=∑ypy∇log⁡qy(0)∇log⁡qy(0)T-\sum_y p_y \nabla^2 \log q_y(0) = \sum_y p_y \nabla\log q_y(0) \nabla\log q_y(0)^\mathsf{T}
−∑y∇2qy(0)=Ey∼p[∇log⁡p(y)∇log⁡p(y)T].\begin{aligned} \qquad- \sum_y \nabla^2q_y(0) \\ &= \mathbb{E}_{y\sim p}\left[\nabla\log p(y) \nabla\log p(y)^\mathsf{T}\right]. \end{aligned}

Summing over positions and expanding to second order gives Equation (3). The remainder is O(∥δ∥3)O(\lVert\delta\rVert^3) when the downstream network is three times differentiable in the residual stream, which holds for transformer layers with smooth activations and normalisation away from zero norm.

Both approximations in § 6.2 are standard in the pruning literature. The empirical Fisher replaces the model’s predictions by the observed tokens; the two agree when the model is calibrated on the calibration text. The block-diagonal rank-one form drops the curvature between positions and, within a position, in directions other than gtg_t. We validate the resulting score against the exact repaired loss increases in § 7.3.

Algorithms

We always keep the first layer: the map of a run is folded into the kept layer before the run, so a run that starts at the first layer cannot be repaired. In every enumerated setting the first group is also critical.

Algorithm 1 Greedy selection under repair (RGS)

Require: layers 1..L1..L, budget nn, objective JJ, activations of the intact model

  1. S←∅S \leftarrow\varnothing

  2. for k=1,…,nk = 1, \ldots, n do

  3. for each kept layer i≠1i \ne1 do

  4. fit Ta,bT_{a,b} by (1) for each run a..ba..b of S∪{i}S \cup\{i\} ▷ cached

  5. Ji←J(model without S∪{i}, repaired)J_i \leftarrow J(\text{model without } S \cup\{i\}, \text{ repaired})

  6. end for

  7. S←S∪{arg min⁡iJi}S \leftarrow S \cup\{\operatorname{arg\,min}_i J_i\}

  8. end for

  9. return SS, with its maps folded into the down-projections

Mechanism of local super-additivity

Table Table 5 lists the explanations we tested. Only functional similarity has partial support: given the distance, the linear CKA of the updates of two groups also predicts their pairwise term on Qwen3-1.7B and Llama, weakly on Qwen3-14B, and not on Gemma.

ExplanationTestResult
Distribution shift of the residual streamreplace removed groups by their mean updateruled out: pairwise terms remain (+0.55 against +0.38 for neighbours)
Later layers restore the updateshare of the removed update that later layers restoreruled out: Spearman +0.17 with the pairwise terms, against +0.72 for distance
Convexity of the losssecond-order split of each pairwise termruled out: curvature term +0.24 at long distance, where the pairwise term is zero
Logit effects addexact split into loss and network partsruled out: joint logit changes differ from the sum by 50% to 70%
Nearby layers compute similar functionspartial Spearman of linear CKA and the pairwise term, given distanceQwen3-1.7B +0.37 (pp = 0.006), Llama +0.36 (0.001), Qwen3-14B +0.11 (0.09), Gemma −0.06 (0.68)

Table 5. Tests of explanations for local super-additivity.

Search without repair

Over the 20 exactly enumerated budgets with at least two removed groups, SLEB finds the optimum in 13 (Figure 7). In 3 of its 7 misses its subset is one shift of one removed group away from the optimum: the optimum keeps a group between two removed runs that greedy removal took out early. A refinement (iterative removal over non-critical groups followed by shift moves, at SLEB’s cost), designed on Qwen3-1.7B, finds the optimum in 16 budgets in every seed and in 2 more in 6% and 68% of seeds. It improves on SLEB on Gemma; on Qwen3-14B at 3 and 4 removed groups SLEB’s subset is already a local optimum under single swaps.

Calibration perplexity relative to the exact optimum for Qwen3-1.7B, Gemma-3-4B, Llama-3.1-8B, and Qwen3-14B

Figure 7. Calibration perplexity of each search method relative to the exact optimum without repair, at every exactly enumerated budget; randomised methods are averaged over 100 seeds.

Additional results

MMLU. Table 12 gives 5-shot MMLU accuracy for every subset. The MMLU objective uses 5 questions per subject from the validation split, 285 in total; the test questions and the dev split, which supplies the 5-shot examples, are disjoint from them. With it RGS is the most accurate method at 7 of 8 settings, by 0.5 to 26.6 points. The exception is Qwen3-1.7B at 7 layers, where ShortGPT reaches 36.1% and RGS 31.3%. On Qwen3-1.7B at 10 and Gemma at 13 layers all methods are within 3 points of chance. Each objective favours the metric that it optimises: the MMLU subsets have 1.7 to 4.9 times the WikiText perplexity of the WikiText subsets, and on Qwen3-14B a five-task accuracy 3 to 10 points lower. At 25% removed, the WikiText subsets of RGS and SLEB on Llama and Qwen3-14B put 0.80 to 0.93 of the probability on the four answer letters, close to the full model’s 0.999, and each answers its most frequent letter for at most 56% of the questions, so they follow the answer format. ReplaceMe’s block on Qwen3-14B at 10 layers puts 0.06 on the letters and answers A for 80% of the questions. On Qwen3.8-27B the task objective gives the highest MMLU accuracy at both budgets: 57.6% at 16 layers against 54.8% for SLEB and 24.1% for ReplaceMe, and 31.2% at 24 layers against 30.2% for SLEB. The gain of the task objective thus carries over from the zero-shot tasks to knowledge questions on the largest model.

Selection time. On Llama-3.1-8B at 12 of 32 layers on one RTX A6000, the selection takes 384 seconds for RGS (319 pruned models), 343 for SLEB with repair, 487 for LaCo, 56 for Prune&Comp, 33 for RFS, 8 for ReplaceMe and 1 for ShortGPT. Each repaired method also needs 29 seconds to collect the activations for the maps.

Calibration size. With 16 calibration sequences the optimum on the calibration sequences is also the optimum on the test sequences in 13 of 17 enumerated settings. Resampling calibration sets from a pool of 64 shows that the disagreement is sampling noise: agreement rises to 100% at 64 sequences on Qwen3-1.7B and Gemma-3-4B.

Repair fitting. Two schemes that fit later maps on the already repaired network did not improve on ReplaceMe’s map. Fitting towards the activations of the intact network made multi-run subsets 1% to 16% worse on Qwen3-1.7B and Llama; fitting towards the output of the removed run, computed on the inputs of the repaired network, was about 5% better on Qwen3-1.7B and within 1.5% either way on Llama. These differences are small compared with the differences between selection methods in Table 3, so the gains of RGS come from the choice of layers.

Per-task results

Per-task accuracy shows where the WikiText objective and task accuracy disagree. At 25% removed, ShortGPT’s and ReplaceMe’s deep blocks keep WinoGrande on Llama (71.1 and 69.7 against 54.4 for RGS), while RGS with the WikiText objective removes middle layers that WinoGrande needs but that have little effect on perplexity. The task objective recovers this: RGS-T reaches 67.6 on WinoGrande there, has the best ARC-Easy accuracy on all four families at both budgets, and the best ARC-Challenge accuracy at 6 of these 8 settings.

Implementation details

Models run in bfloat16. Sequences begin with a BOS token when the tokenizer adds one. Repair maps are solved in float32 with the ridge; without a ridge they are solved in float64 with a pseudo-inverse, because Gemma’s MLP outputs on SlimOrca are rank-deficient. For Gemma 3 the map acts on the output of the post-feedforward normalisation, which cannot be folded into a weight, so the repaired Gemma model has one extra d×dd \times d

matrix per run. Qwen3.8-27B runs split over two GPUs. During the study we found and fixed two faults, each recorded with its effect in the repository: removed or moved Gemma layers received the wrong attention type, and unregularised maps were first solved by a routine that assumes full rank. Every number in this paper comes from runs after the fixes. All data behind the tables and figures is exported to CSV files in the supplementary material.

ModelnnReplaceMeReplaceMe, task dataRGS-T
Qwen3-1.7B745.643.847.1
1040.940.941.7
Gemma-3-4B850.845.859.9
1341.540.651.2
Llama-3.1-8B857.258.961.2
1246.846.851.9
Qwen3-14B1051.451.459.5
1544.538.847.9

Table 8. Average accuracy (%) with the task objective, against ReplaceMe’s criterion computed on the hidden states of the same task training texts.

ModelnnC4 PPLWikiText PPLAvg
Qwen3-1.7B725.5 / 25.029.3 / 28.852.0 / 50.8
1031.0 / 30.734.4 / 34.247.1 / 47.3
Gemma-3-4B820.1 / 19.627.4 / 25.656.1 / 56.9
1331.2 / 25.543.9 / 35.448.5 / 51.2
Llama-3.1-8B821.9 / 18.329.2 / 21.362.7 / 58.6
1230.7 / 22.939.4 / 27.456.2 / 54.4
Qwen3-14B1018.8 / 17.319.9 / 17.359.6 / 60.2
1522.2 / 22.722.9 / 23.651.7 / 54.1

Table 10. After 200 LoRA steps on C4, mean of three seeds: ReplaceMe / RGS. The spread between seeds is 0.01 to 0.3 in C4 perplexity, 0.06 to 0.66 in WikiText perplexity and at most 0.5 points in accuracy.

References

  1. [1]Ali Bahri, Hang Li, Hongliang Li, and Zhitang Chen. 2026. SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs. CoRR, abs/2608.25068.arxiv.org/abs/2608.25068
  2. [2]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020., pages 7432–7439. AAAI Press.arxiv.org/abs/1911.11641
  3. [3]Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. 2025a. Streamlining Redundant Layers to Compress Large Language Models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.arxiv.org/abs/2403.19135
  4. [4]Xinrui Chen, Haoli Bai, Tao Yuan, Ruikang Liu, Kang Zhao, Xianzhi Yu, Lu Hou, Tian Guan, Yonghong He, and Chun Yuan. 2025b. A Simple Linear Patch Revives Layer-Pruned Large Language Models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.arxiv.org/abs/2505.24680
  5. [7]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2924–2936. Association for Computational Linguistics.arxiv.org/abs/1905.10044
  6. [8]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. CoRR, abs/1803.05457.arxiv.org/abs/1803.05457
  7. [9]Elia Cunegatti, Marcus Vukovjevic, Erik Nielsen, and Giovanni Iacca. 2026. From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression. CoRR, abs/2606.02559.arxiv.org/abs/2606.02559
  8. [10]Xuan Ding, Pengyu Tong, Ranjie Duan, Yunjian Zhang, Rui Sun, and Yao Zhu. 2026. Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models. In The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc.
  9. [11]Elias Frantar and Dan Alistarh. 2022. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.arxiv.org/abs/2208.11580
  10. [12]Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA., volume 202 of Proceedings of Machine Learning Research, pages 10323–10337. PMLR.arxiv.org/abs/2301.00774
  11. [13]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.arxiv.org/abs/2210.17323
  12. [15]Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. 2025. The Unreasonable Ineffectiveness of the Deeper Layers. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.arxiv.org/abs/2403.17887
  13. [16]Babak Hassibi and David G. Stork. 1993. Second Order Derivatives for Network Pruning: Optimal Brain Surgeon. In Advances in Neural Information Processing Systems 5, [NIPS Conference, Denver, Colorado, USA, November 30 - December 3, 1992], pages 164–171. Morgan Kaufmann.
  14. [17]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  15. [18]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yunzh i Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.arxiv.org/abs/2106.09685
  16. [19]Bo-Kyeong Kim, Geon min Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. 2024. Shortened LLaMA: A Simple Depth Pruning for Large Language Models. CoRR, abs/2402.02834.
  17. [20]Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim, Youngjin Heo, Suin Cho, Seong hun Kim, Woosang Lim, and Gaeul Kwon. 2026. Rethinking Layer Redundancy in Large Language Models: Calibration Objectives and Search for Depth Pruning. CoRR, abs/2604.24938. Cited version: arXiv:2604.24938v1. DBLP metadata corresponds to this version.
  18. [21]Yann LeCun, John S. Denker, and Sara A. Solla. 1990. Optimal Brain Damage. In Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989], pages 598–605. Morgan Kaufmann.
  19. [22]Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.arxiv.org/abs/2102.05426
  20. [23]Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanviet Yong, and Teknium. 2023. SlimOrca: An Open Dataset of GPT-4 Augmented FLAN Reasoning Traces, with Verification.
  21. [24]Kainan Liu, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, and Jing Xiao. 2025. GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 26333–26348. Association for Computational Linguistics.arxiv.org/abs/2501.00339
  22. [25]Llama Team. 2024. The Llama 3 Herd of Models. CoRR, abs/2407.21783.
  23. [26]Yao Lu, Yuqi Li, Wenbin Xie, Shanqing Yu, Qi Xuan, Zhaowei Zhu, and Shiping Wen. 2025. The Structural Scalpel: Automated Contiguous Layer Pruning for Large Language Models. CoRR, abs/2510.23652.arxiv.org/abs/2510.23652
  24. [27]Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2025. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 20192–20204. Association for Computational Linguistics.
  25. [28]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.arxiv.org/abs/1609.07843
  26. [29]Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance Estimation for Neural Network Pruning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 11264–11272. Computer Vision Foundation / IEEE.arxiv.org/abs/1906.10771
  27. [30]B. Moore. 1981. Principal component analysis in linear systems: Controllability, observability, and model reduction. IEEE Transactions on Automatic Control, 26(1):17–32.DOI
  28. [32]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21:140:1–140:67.arxiv.org/abs/1910.10683
  29. [33]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732–8740. AAAI Press. Dmitriy Shopkhoev, Ammar Ali, Magauiya Zhussip, Valentin Malykh, Stamatios Lefkimmatis, Nikos Komodakis, and Sergey Zagoruyko. 2025. ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.
  30. [34]Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas M. Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov. 2024. A deeper look at depth pruning of LLMs. CoRR, abs/2407.16286.
  31. [35]Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, and Dan Alistarh. 2025. EvoPress: Accurate Dynamic Model Compression via Evolutionary Search. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net.arxiv.org/abs/2410.14649
  32. [36]Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. 2024. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 46136–46155. PMLR / OpenReview.net.arxiv.org/abs/2402.09025
  33. [37]Lintang Sutawika, Leo Gao, Hailey Schoelkopf, Stella Biderman, Jonathan Tow, Baber Abbasi, ben fattori, Charles Lovering, farzanehkhaee70, Jason Phang, Anish Thite, Fazz, Aflah, Niklas Muennighoff, Thomas Wang, sdtblck, nopperl, gakada, tttyuntian, and 11 others. 2023. EleutherAI/lm-evaluation-harness: Major refactor. Software, version v0.4.0.DOI
  34. [38]Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. 2018. Faster gaze prediction with dense networks and Fisher pruning. CoRR, abs/1801.05787.arxiv.org/abs/1801.05787
  35. [39]Yifei Yang, Zouying Cao, and Hai Zhao. 2024. LaCo: Large Language Model Pruning via Layer Collapse. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, volume EMNLP 2024 of Findings of ACL, pages 6041–6417. Association for Computational Linguistics.arxiv.org/abs/2402.11187
  36. [40]Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, and Sunwoo Lee. 2026. Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs. CoRR, abs/2605.15491.arxiv.org/abs/2605.15491
  37. [41]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics.arxiv.org/abs/1905.07830
  38. [42]Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. 2025. BlockPruner: Fine-grained Pruning for Large Language Models. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 5065–5080. Association for Computational Linguistics.arxiv.org/abs/2406.10594

Paper details

vlnc:W_e8ktdthj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents