WorksComputer Science
Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models
Abstract
Depth pruning removes entire transformer layers to make large language models faster, and recent methods recover quality by replacing the removed layers with a training-free linear map. Which layers to remove, however, is chosen with heuristic scores whose distance from the best choice is unknown. We evaluate every subset of layer groups exactly (about 53,000 pruned models across four families, with and without repair) and find that the best layers to remove depend on the repair: linear repair absorbs the interaction among nearby layers and turns the optimum from a split layout into a contiguous block, so methods that choose layers without the repair miss the repaired optimum by up to 189%. We therefore select layers under the repair that will be applied. A greedy search at the cost of SLEB lowers perplexity by 20% to 66% relative to ReplaceMe on five models up to 27B parameters, and with a task objective it is up to 25 points more accurate given the same data. A second-order expansion of the loss yields a score that selects the repaired block from one forward and one backward pass of the intact model.
Introduction
Depth pruning removes whole transformer layers. Unlike width pruning or sparsity, it reduces latency and memory on standard hardware without special kernels, because the pruned model is a shorter model of the same architecture [27, 36, 19]. Recent methods add a training-free repair: they replace the removed layers by a linear map fitted on calibration activations and fold the map into the weights of a kept layer, so the repaired model has the same parameter count as the pruned one [4] (Shopkhoev et al. 2025; Chen et al. 2026).
Every depth-pruning method must choose which layers to remove. For layers and removed layers there are choices, and each choice must be scored on a pruned model. Published methods therefore use proxies: the similarity of a layer’s input and output [27, 15] (Shopkhoev et al. 2025), greedy removal [36], evolutionary search [35], or game-theoretic estimates of each layer’s contribution [34, 10]. How far these choices are from the best one is unknown, and recent work concludes that more elaborate search adds little [20].
These proxies score each layer in the unrepaired model. With a linear repair, the interactions among the removed layers change: a map absorbs the joint effect of contiguous layers, so the layers that are best to remove together differ from those individually least important. The gap between the proxy and the true optimum therefore depends on the repair, and it is unknown because the true optimum has not been computed. We close this gap. At a granularity of two-layer groups the number of subsets is small enough to evaluate every one: about 53,000 pruned models over Qwen3-1.7B, Gemma-3-4B, Llama-3.1-8B and Qwen3-14B, each scored after plain removal and after the least-squares repair of ReplaceMe (Shopkhoev et al., 2025). The exact optima show that the answer depends on the repair (Figure 1): without repair, removing two nearby layers costs more than the sum of their individual costs, so the best subset spreads the removal over separate runs and a contiguous block is close to the worst choice. With repair, one map absorbs the interaction inside a run, and the best subset becomes a block, or a block and one short run. Published methods choose layers without the repair, and so miss the repaired optimum by up to 189%.

Figure 1. Repair changes which set of removed layers is best. Perplexity of the best contiguous block relative to the exact optimum over all subsets, at every enumerated budget of four models. Without repair (grey) a block is up to 6.9 times worse than the optimum. With ReplaceMe’s linear repair (red) the best block is the optimum at 9 of the 18 budgets and within 5% of it at 13.
The fix follows directly: evaluate every candidate subset on the repaired model that will be deployed (Figure 2). A greedy search that does so costs as much as SLEB and finds the exact repaired optimum, or comes within 0.2% of it, at every enumerated budget on three of four models. A second-order expansion of the repaired loss turns the same principle into a score computed from one forward and one backward pass of the intact model. Our contributions are:

Figure 2. Overview. (a) Evaluating every subset of layer groups shows critical layers and local super-additivity, so the optimum without repair splits the removal into separate runs. (b) ReplaceMe’s least-squares map replaces a run by a linear map folded into the down-projection of the kept layer before it; it absorbs the interaction inside the run, and the repaired optimum becomes one block. (c) RGS evaluates every candidate on its repaired model at the cost of SLEB; RFS ranks blocks by the Fisher-weighted residual of their repair, computed on the intact model only.
The structure of the exact optimum. A Möbius decomposition of the loss increase from removal shows two interactions in all four families: a few critical layers whose joint loss increase is far below the sum of their individual loss increases, and local super-additivity among all other layers, which decays with distance in depth with a length of 3 to 6 layers (§ 4). As a result, at 4 to 6 removed groups the best contiguous block is 18% to 588% worse than the optimum.
Linear repair makes the optimum contiguous. One least-squares map per run absorbs the interaction among the run’s layers, and the repaired optimum becomes one block, or one block and one short run (§ 5). Selection must therefore account for the repair: choosing layers without the repair and then repairing them is 5% to 33% worse than the repaired optimum, and ReplaceMe’s cosine criterion is 11% to 189% worse.
Selection methods. Greedy selection under repair (RGS), at the cost of SLEB, has 20% to 66% lower perplexity than ReplaceMe on four models and 54% to 56% lower on a 27B hybrid-attention model, and with a task objective is more accurate at every budget, by up to 25 points on the 27B model. A second-order expansion of the Kullback–Leibler divergence gives a score (RFS) that selects the repaired block from the gradients of the intact model (§ 6).
Evaluation beyond the selection data. We test the methods with ReplaceMe’s default calibration, on tasks that the objective does not see, on C4, after LoRA recovery, and on a 27B
model that was not used for any design choice, and we report the cases where a baseline is better (§ 7).
Related work
Choosing layers to remove. ShortGPT removes the layers whose output is most similar to their input [27]; Gromov et al. (2025) remove the contiguous block with the smallest angular distance between its input and output, and LLM-Streamline [3], LaCo [39] and CLP [26] remove or merge contiguous blocks by design. SLEB removes layers greedily by the perplexity of the pruned model [36]; BlockPruner does the same for attention and MLP sublayers [42]. EvoPress searches over subsets and observes that removal is not additive [35], and Shapley-value estimates model the interaction with sampling or surrogates [34, 10]. Kim et al. (2026) call the subset problem NP-hard and find that search gains little over one-shot scores without repair. We compute the optimum exactly, decompose its interactions, and show that the gain of search depends on the repair.
Training-free repair. ReplaceMe fits one linear map per removed block by least squares and folds it into the down-projection of the last kept layer (Shopkhoev et al., 2025). LinearPatch [4] and Prune&Comp (Chen et al., 2026) correct scale and outliers at the pruning interface; Prune&Comp alternates pruning and compensation, so each candidate is scored on a model that earlier steps have compensated, with one scalar per layer. Other repairs use low-rank or shifted maps [24, 40, 1], and SubFit selects submodules by the residual of their fitted replacement [9]. These methods repair after the layers are chosen, or choose with a criterion that does not evaluate the repaired model. ReplaceMe reports that its cosine criterion finds optimal or near-optimal 8-layer blocks on Llama 3 8B (Shopkhoev et al., 2025, App. A.10); over all block lengths and four families we find it 11% to 189% worse in perplexity than the repaired optimum.
Second-order saliency and model reduction. Optimal Brain Damage and Optimal Brain Surgeon score the removal of a weight by a second-order expansion of the loss [21, 16], and OBC, GPTQ and SparseGPT apply it layer by layer [11, 13, 12]. Fisher pruning and Taylor importance score the removal of an activation by [38, 29], and BRECQ weights reconstruction error by the Fisher information [22]. Our score applies this expansion to the error that remains after repair. Balanced truncation reduces a linear system by weighting the error of the reduced model by how much the output observes it [30]; our score has the same form, with the loss gradient in the role of the observability.
Problem setup
Pruning and repair. A decoder-only model with layers updates a residual stream at each token by . We partition the layers into contiguous groups of one or two layers. Removing a set of groups skips their layers; a run is a maximal set of adjacent removed groups. Linear repair (Shopkhoev et al., 2025) replaces a run of layers by a map on the MLP output of the last kept layer before the run, which then outputs instead of , with . The map solves
where the rows of , and are activations of the intact model at calibration tokens. folds into the down-projection of layer , so the repaired model has the architecture of the pruned one. Each run has its own map; a run that starts at the first layer cannot be repaired.
Objective and data. The loss increase of a subset is the mean token loss of the pruned model minus that of the intact model on 16 WikiText-2 calibration sequences of 256 tokens [28]. Every method selects layers on these sequences, and we report perplexity (PPL) on 16 other sequences of the WikiText-2 test split. Maps are fitted on 128 WikiText-2 training sequences with times the mean diagonal of . Accuracy is the mean over ARC-Easy, ARC-Challenge, HellaSwag, PIQA and WinoGrande [8, 41, 2, 33], 1,000 items each, with the prompts of lm-evaluation-harness (Sutawika et al., [37]); one standard error of the mean is about 0.7 points. BoolQ (Clark et al., [7]) is reported separately, because pruning shifts the yes/no prior and moves it by 20 points or more independently of the other tasks.
Exact enumeration. We use Qwen3-1.7B and Qwen3-14B (Qwen Team, 2025), Gemma-3-4B (Gemma Team, 2025) and Llama-3.1-8B-Instruct (Llama Team, [25]) with two-layer groups (14, 17, 16 and 20 groups), and evaluate every subset of up to 6 removed groups (4 on Qwen3-14B), without and with repair (Table 14). We also enumerate Qwen3-1.7B at single-layer granularity (28 groups) up to 3 removed layers; the same interaction structure appears at both granularities. We store the loss of every subset on every sequence, so any search method can be evaluated against the exact optimum without further model runs.
| Model | Layers | Groups | Subsets per budget |
| Qwen3-1.7B | 28 | 14 | all, up to 6 removed |
| Qwen3-1.7B | 28 | 28 | all, up to 3 removed |
| Gemma-3-4B | 34 | 17 | all, up to 6 removed |
| Llama-3.1-8B | 32 | 16 | all, up to 6 removed |
| Qwen3-14B | 40 | 20 | all, up to 4 removed |
Table 14. Exactly enumerated settings: every subset of groups up to the stated number of removed groups is evaluated on every calibration and test sequence, without and with repair; about 53,000 subsets in total.
The structure of the exact optimum
Interaction decomposition. Let be the loss increase from removing , with . Its Möbius transform (Grabisch and Roubens, 1999)
splits the loss increase into an individual term for each group and an interaction term for each larger set; removal is additive exactly when every interaction term is zero. The interaction share of a budget, the mean absolute difference between and as a share of the mean absolute loss increase, rises from 21% to 50% at 2 to 6 removed groups on Qwen3-1.7B, 15% to 31% on Gemma, 7% to 25% on Llama and 11% to 22% (2 to 4 removed) on Qwen3-14B.
Critical layers. We call a group critical when its removal alone more than doubles perplexity: 4 of 14 groups in Qwen3-1.7B, 5 of 17 in Gemma, 3 of 16 in Llama and 3 of 20 in Qwen3-14B, usually the first ones. Their pairwise terms are large and negative, because the loss saturates once the model fails, and every exact optimum keeps them. We call the other groups non-critical.
Local super-additivity. Among non-critical groups the pairwise term is positive and decays with the distance in depth (Figure 3), with fitted decay lengths of 2.9 layers (Gemma)

Figure 3. Mean pairwise interaction term of two removed non-critical groups against their distance in depth, with exponential fits over the first four distances (dashed). Removing two nearby groups costs more than the sum of their individual costs.
to 6.4 layers (Llama) and terms close to zero beyond about 10 layers. Pairwise terms explain 52% to 76% of the non-additivity at 3 removed groups; higher-order terms among nearby groups grow at heavier budgets.
A contiguous block is close to the worst choice. A block contains the most pairs within the interaction range, so under local super-additivity it has the largest interaction cost. At small budgets the optimum can be a block; from 4 removed groups on, every exact optimum in every family splits the removal into 2 or 3 runs, and the best block is 18% to 588% worse in perplexity (Table 1). Methods that remove one block by design (Gromov et al., [15]; Chen et al., [3]; Lu et al., [26]) therefore cannot find the optimum when the layers are not repaired.
| Model | 3 | 4 | 5 | 6 |
| Qwen3-1.7B | 0% | 28% | 77% | 588% |
| Gemma-3-4B | 64% | 252% | 512% | 447% |
| Llama-3.1-8B | 32% | 34% | 76% | 190% |
| Qwen3-14B | 0% | 18% | – | – |
Table 1. Excess perplexity of the best contiguous block over the exact optimum without repair.
Implications for existing methods. One-shot scoring removes the groups with the smallest individual loss increase. These cluster in depth, so one-shot scoring removes an almost contiguous set: 33% above the optimum on Qwen3-1.7B and 581% on Gemma at 4 removed groups. A pairwise model assigns large negative terms to critical-layer pairs; its optimum is 189 to 226,000 times worse. SLEB evaluates actual pruned models and accounts for the interaction, but cannot undo an earlier removal: it finds the optimum in 13 of 20 budgets (Section D). The interaction remains when a removed layer is replaced by its mean update; it therefore comes from the lost computation (Section C).
Linear repair makes the optimum contiguous
Table 2 compares the exact optimum without repair, that optimum after repair, ReplaceMe’s choice, and the exact optimum under repair over every subset of non-critical groups.
| Model | Removal opt. as is | Removal opt. repaired | ReplaceMe | Repaired opt. | |
| Qwen3-1.7B (26.5) | 4 | 73.4 | 50.3 | 75.1 | 41.7 |
| Qwen3-1.7B (26.5) | 5 | 128.7 | 59.2 | 90.6 | 54.3 |
| Qwen3-1.7B (26.5) | 6 | 229.2 | 80.2 | 99.5 | 76.0 |
| Gemma-3-4B (32.0) | 4 | 46.7 | 50.6 | 44.7 | 38.3 |
| Gemma-3-4B (32.0) | 5 | 65.9 | 51.6 | 72.2 | 42.8 |
| Gemma-3-4B (32.0) | 6 | 93.5 | 64.1 | 149.8 | 51.9 |
| Llama-3.1-8B (13.7) | 4 | 33.6 | 26.8 | 40.4 | 22.2 |
| Llama-3.1-8B (13.7) | 5 | 53.0 | 32.5 | 61.9 | 28.3 |
| Llama-3.1-8B (13.7) | 6 | 84.2 | 44.1 | 94.0 | 35.1 |
| Qwen3-14B (15.0) | 4 | 23.1 | 25.6 | 21.3 | 19.1 |
Table 2. Perplexity at the heaviest enumerated budgets ( removed groups; full model in parentheses). “Removal opt., repaired” repairs the best subset for plain removal. The last column is the exact optimum under repair.
Repair absorbs the interaction inside a run. One map is fitted to the input and output of a whole run, so the super-additivity among the run’s layers is part of what it fits. At the heaviest budget the interaction share falls from 62% to 36% on Qwen3-1.7B, 66% to 52% on Gemma, 57% to 28% on Llama and 25% to 15% on Qwen3-14B. Separately repaired runs still interact: the sum of their repaired loss increases explains the loss increase of a subset with of 0.8 to 0.9 at 2 removed groups but only to 0.44 at 6, because each map assumes that the rest of the network is intact.
The repaired optimum is contiguous. Without repair, every optimum at 4 or more removed groups splits into 2 or 3 runs. Under repair it is one block (Qwen3-1.7B at 4 and 5 removed groups, Qwen3-14B at 4) or one block and one more run (Figure 8 in Section E). A contiguous block, which is close to the worst choice without repair, is close to the best choice with repair.
Figure 8. Groups removed by the exact optimum at the heaviest enumerated budget, without repair (grey) and with repair (red); light cells are critical groups. Repair turns two or three separate runs into one block, or a block and a short run, at 17% to 67% lower perplexity.
Selection must account for the repair. Two simpler approaches are worse. Repairing the best subset for plain removal is 5% to 33% worse than the repaired optimum, because that subset is split into runs that repair cannot absorb. Choosing the block with the smallest cosine distance between its input and output, as ReplaceMe does, is 11% to 189% worse. On Llama at 8 of 32 layers, a setting that ReplaceMe evaluates, its block is among the most accurate subsets under the WikiText objective (Table 3), in agreement with Shopkhoev et al. (2025, App. A.10), but its perplexity is 79% higher than that of the subset chosen under repair, and with a task objective the subset chosen under repair is 4.0 points more accurate. A selection method must therefore evaluate each subset on the repaired model that it deploys.
| Qwen3-1.7B 7/28 PPL↓ | Qwen3-1.7B 7/28 Avg↑ | Qwen3-1.7B 10/28 PPL↓ | Qwen3-1.7B 10/28 Avg↑ | Gemma-3-4B 8/34 PPL↓ | Gemma-3-4B 8/34 Avg↑ | Gemma-3-4B 13/34 PPL↓ | Gemma-3-4B 13/34 Avg↑ | Llama-3.1-8B 8/32 PPL↓ | Llama-3.1-8B 8/32 Avg↑ | Llama-3.1-8B 12/32 PPL↓ | Llama-3.1-8B 12/32 Avg↑ | Qwen3-14B 10/40 PPL↓ | Qwen3-14B 10/40 Avg↑ | Qwen3-14B 15/40 PPL↓ | Qwen3-14B 15/40 Avg↑ | Qwen3.8-27B 16/64 PPL↓ | Qwen3.8-27B 16/64 Avg↑ | Qwen3.8-27B 24/64 PPL↓ | Qwen3.8-27B 24/64 Avg↑ | |
| Full model | 26.5 | 60.0 | 26.5 | 60.0 | 32.0 | 69.8 | 32.0 | 69.8 | 13.7 | 72.4 | 13.7 | 72.4 | 15.0 | 73.0 | 15.0 | 73.0 | 10.1 | 72.8 | 10.1 | 72.8 |
| Same calibration text for every method (WikiText-2) | ||||||||||||||||||||
| ShortGPT + R | 77.4 | 46.1 | 130.0 | 40.4 | 44.7 | 50.8 | 87.2 | 42.7 | 39.4 | 59.5 | 94.0 | 46.8 | 48.2 | 49.2 | 266.4 | 37.7 | 34.8 | 40.6 | 42.4 | 38.6 |
| LaCo + R | 78.7 | 39.9 | 571.3 | 36.2 | 57.0 | 54.0 | 445.4 | 39.6 | 46.7 | 58.5 | 82.6 | 45.6 | 90.9 | 41.1 | 1119.1 | 36.4 | n/a | n/a | n/a | n/a |
| ReplaceMe | 53.7 | 45.6 | 68.7 | 40.9 | 44.7 | 50.8 | 150.7 | 41.5 | 40.4 | 57.2 | 94.0 | 46.8 | 41.9 | 51.4 | 43.3 | 44.5 | 28.7 | 42.2 | 36.9 | 42.2 |
| Prune&Comp + R | 63.5 | 46.7 | 120.2 | 41.2 | 44.7 | 50.8 | 192.3 | 40.6 | 42.2 | 57.0 | 87.0 | 45.6 | 45.4 | 51.4 | 76.6 | 42.2 | 28.1 | 48.9 | 33.9 | 42.9 |
| SLEB + R | 40.3 | 45.9 | 59.1 | 41.8 | 50.5 | 54.7 | 69.1 | 44.7 | 23.6 | 56.2 | 37.8 | 48.4 | 27.2 | 49.1 | 43.1 | 43.0 | 13.2 | 65.7 | 17.1 | 57.4 |
| RFS (ours) | 38.7 | 44.9 | 54.3 | 41.1 | 38.2 | 48.1 | 97.6 | 40.6 | 24.3 | 51.2 | 95.9 | 39.4 | 21.9 | 53.0 | 34.0 | 45.8 | 13.4 | 66.4 | 15.7 | 55.5 |
| RGS (ours) | 37.7 | 44.2 | 54.3 | 41.1 | 35.9 | 50.6 | 51.1 | 44.7 | 22.6 | 55.4 | 34.4 | 49.0 | 21.9 | 54.4 | 34.1 | 45.1 | 13.3 | 64.9 | 16.0 | 57.9 |
| RGS vs. ReplaceMe | −30% | −1.5 | −21% | +0.2 | −20% | −0.2 | −66% | +3.2 | −44% | −1.8 | −63% | +2.2 | −48% | +3.0 | −21% | +0.6 | −54% | +22.7 | −56% | +15.7 |
| Selection on task training data† | ||||||||||||||||||||
| ReplaceMe, task data | 51.9 | 43.8 | 68.7 | 40.9 | 71.3 | 45.8 | 192.3 | 40.6 | 40.9 | 58.9 | 94.0 | 46.8 | 41.9 | 51.4 | 251.3 | 38.8 | 28.7 | 42.2 | 36.9 | 42.2 |
| RGS-T (ours) | 64.5 | 47.1 | 110.0 | 41.7 | 68.1 | 59.9 | 143.7 | 51.2 | 34.6 | 61.2 | 54.0 | 51.9 | 37.9 | 59.5 | 67.7 | 47.9 | 27.6 | 66.8 | 33.3 | 59.8 |
| RGS-T vs. ReplaceMe, task data | +24% | +3.3 | +60% | +0.9 | −5% | +14.1 | −25% | +10.6 | −15% | +2.3 | −43% | +5.1 | −10% | +8.1 | −73% | +9.1 | −4% | +24.6 | −10% | +17.6 |
Table 3. Main results: WikiText-2 perplexity (PPL) and the average zero-shot accuracy (%) over ARC-Easy, ARC-Challenge, HellaSwag, PIQA and WinoGrande (Avg; one standard error is about 0.7 points). Every method is repaired with the same least-squares maps (+R); the methods differ only in which layers they remove. The upper block selects on WikiText-2 calibration text; the lower block (†) selects on the training items of the five tasks, which are disjoint from the evaluation items. Within each block the best result is highlighted and the second best underlined; italic rows give the change relative to ReplaceMe. LaCo’s merge needs identical layers and does not apply to the hybrid 27B model. Per-task scores are in Table 13.
Selecting layers under repair
Greedy selection under repair
We therefore include the repair in the search. Greedy selection under repair (RGS) removes one layer at a time, as SLEB does, but evaluates each candidate subset after fitting and applying its repair maps (Algorithm 1). Maps are fitted once per run from cached activations of the intact model, so a candidate costs one forward pass over the calibration text and RGS costs as many evaluations as SLEB, for removed layers. On the exact tables RGS finds the repaired optimum at every budget on Qwen3-1.7B and Qwen3-14B, and is within 0.2% of it on Llama and within 3.4% on Gemma. The objective can be any loss on the repaired model. With a task objective, the cross-entropy of the correct choice among the choices on 200 training items per task (RGS-T), it selects for accuracy.
A second-order score from the intact model
Under repair the optimum is close to one block, so ranking blocks is sufficient, and a second-order expansion of the loss ranks them with the activations and gradients of the intact model. Let be the residual stream after block at the calibration tokens of the intact model. The repaired model produces there, with the residual of the repair at token ; the layers after the block are unchanged.
Proposition 1. Let be the next-token distribution at position as a function of the residual stream, and . Then , and
where is the Fisher information of the downstream network with respect to the residual stream.
The proof is in Section A. We replace the expectation over by the observed tokens (the empirical Fisher) and keep one rank-one block per position, with the gradient of the summed token loss, so that each position plays the role of a sample as in Fisher pruning (Theis et al., [38]) and Taylor importance (Molchanov et al., [29]). The result is the repair Fisher score (RFS) of a block:
The least-squares map of Equation (1) minimises the Euclidean norm of ; Equation (4) measures the same residual in the metric of the downstream Fisher information, which weights each direction of the residual stream by how much the loss responds to it. This is the weighting that balanced truncation applies to the error of a reduced system, with the observability Gramian in the role of (Moore, [30]). The unweighted residual and the first-order term give inconsistent block rankings across models (§ 7.3).
Experiments
Experimental setup
Models and budgets. We use the four enumerated families of § 3 and Qwen3.8-27B, which has 64 layers with three linear-attention layers to each full-attention layer and was not used for any design choice. We remove 25% and 37.5% of the layers at single-layer granularity.
Baselines. ShortGPT’s block influence (Men et al., [27]), LaCo (Yang et al., [39]), ReplaceMe’s cosine block (Shopkhoev et al., 2025), Prune&Comp (Chen et al., 2026) and SLEB (Song et al., [36]). Every method is repaired with the same least-squares maps (+R), so the methods differ only in which layers they choose. For LaCo we remove the layers that its merge criterion selects, because this is better than its parameter merge at every budget we test. For the task objective we also give ReplaceMe’s criterion the same task training texts.
Metrics and implementation. We report WikiText-2 perplexity on the test sequences, C4 perplexity, and the average accuracy over five zero-shot tasks (§ 3). All models run in bfloat16 on one 48 GB GPU, except Qwen3.8-27B, which runs on two (Section G).
Main results
Perplexity. On the four enumerated families RGS has the lowest perplexity at 7 of 8 settings, with RFS 0.3% closer at the eighth; it is 20% to 66% below ReplaceMe and 4% to 29% below SLEB with repair (Table 3; Figure 9 in Section E). SLEB uses the same greedy search without repair, so this difference comes from the repair in the search. It is 20% to 73% below Prune&Comp and 37% to 97% below LaCo, both with repair, on every model they apply to. The gap to ReplaceMe is largest where the repaired optimum is not one block: on Gemma at 13 of 34 layers RGS removes a block of 8 layers and 5 separate ones and has one third of ReplaceMe’s perplexity. On Qwen3.8-27B it is 54% and 56% below ReplaceMe, whose criterion picks the first layers on this architecture; RGS matches SLEB with repair at 16 layers (within 0.5%) and is 7% better at 24. Across all five models and both budgets, including the repair in the search consistently lowers perplexity.

Figure 9. Perplexity relative to ReplaceMe (log scale), every method repaired by the same maps. RGS is 20% to 66% lower on the four enumerated families and 54% to 56% lower on Qwen3.8-27B.
Accuracy. Given the same task training data, RGS-T is more accurate than ReplaceMe’s criterion at all 8 settings, by 0.9 to 14.1 points (lower block of Table 3, Figure 4), and also has lower perplexity at 6 of them. Both methods use the same data, so the gain comes from selection under repair. RGS-T has the highest average of all methods at 7 of 8 settings and beats plain ReplaceMe at all 8, by 0.9 to 9.8 points; on Gemma at 8 of 34 layers it keeps 86% of the full model’s accuracy against ReplaceMe’s 73%.

Figure 4. Average accuracy over the five zero-shot tasks. Labels give the gain of RGS-T over ReplaceMe’s criterion given the same task data (0.9 to 14.1 points); dashed lines mark the full model.
The accuracy gain transfers to the 27B model and to tasks outside the objective. On Qwen3.8-27B, RGS-T keeps 92% of the full model’s accuracy at 16 of 64 layers and 82% at 24, 24.6 and 17.6 points above ReplaceMe given the same data, and above every other method at both budgets. WikiText perplexity and task accuracy favour different layers (Table 13). Built from ARC-Easy, HellaSwag and PIQA only, the task objective still beats ReplaceMe on ARC-Challenge and WinoGrande at both budgets on three of four models (Table 9). Selection under repair thus improves both perplexity and accuracy, and the gain generalises across models, budgets and tasks.
| Model | ReplaceMe | RGS-T, 3 tasks | RGS-T, 5 tasks | |
| Qwen3-1.7B | 7 | 38.1 | 40.0 | 40.0 |
| 10 | 37.0 | 37.2 | 37.1 | |
| Gemma-3-4B | 8 | 41.0 | 52.1 | 52.6 |
| 13 | 37.3 | 43.2 | 43.6 | |
| Llama-3.1-8B | 8 | 53.4 | 54.3 | 54.3 |
| 12 | 47.9 | 47.3 | 47.3 | |
| Qwen3-14B | 10 | 42.1 | 50.0 | 52.2 |
| 15 | 38.9 | 43.5 | 42.2 |
Table 9. Mean accuracy (%) on ARC-Challenge and WinoGrande when the task objective uses only ARC-Easy, HellaSwag and PIQA (3 tasks); one standard error is about 1.1 points. The highlight compares ReplaceMe with the 3-task objective.
| Method | 25% of layers removed | 37.5% of layers removed | ||||||||||||
| PPL↓ | ARC-e | ARC-c | HS | PIQA | WG | Avg↑ | PPL↓ | ARC-e | ARC-c | HS | PIQA | WG | Avg↑ | |
| Qwen3-1.7B, 28 layers: 7 and 10 removed | ||||||||||||||
| Full model | 26.5 | 70.0 | 42.6 | 53.9 | 72.8 | 60.7 | 60.0 | 26.5 | 70.0 | 42.6 | 53.9 | 72.8 | 60.7 | 60.0 |
| ShortGPT + R | 77.4 | 46.6 | 29.8 | 38.4 | 61.0 | 54.6 | 46.1 | 130.0 | 35.8 | 24.4 | 34.6 | 56.0 | 51.2 | 40.4 |
| ReplaceMe | 53.7 | 47.1 | 25.5 | 42.4 | 62.6 | 50.6 | 45.6 | 68.7 | 34.4 | 22.4 | 38.2 | 57.6 | 51.7 | 40.9 |
| SLEB + R | 40.3 | 42.8 | 26.0 | 43.5 | 64.3 | 52.7 | 45.9 | 59.1 | 36.2 | 22.0 | 40.0 | 58.2 | 52.5 | 41.8 |
| RFS (ours) | 38.7 | 42.4 | 24.3 | 42.7 | 62.0 | 53.0 | 44.9 | 54.3 | 35.8 | 21.4 | 38.6 | 56.7 | 53.0 | 41.1 |
| RGS (ours) | 37.7 | 41.6 | 24.1 | 41.8 | 61.7 | 51.7 | 44.2 | 54.3 | 35.8 | 21.4 | 38.6 | 56.7 | 53.0 | 41.1 |
| RGS-T (ours) | 64.5 | 49.3 | 28.0 | 42.7 | 63.5 | 52.0 | 47.1 | 110.0 | 39.9 | 23.6 | 37.6 | 56.8 | 50.7 | 41.7 |
| Gemma-3-4B, 34 layers: 8 and 13 removed | ||||||||||||||
| Full model | 32.0 | 78.9 | 57.1 | 65.2 | 78.1 | 69.6 | 69.8 | 32.0 | 78.9 | 57.1 | 65.2 | 78.1 | 69.6 | 69.8 |
| ShortGPT + R | 44.7 | 53.6 | 24.6 | 52.2 | 66.3 | 57.5 | 50.8 | 87.2 | 40.2 | 22.9 | 41.8 | 58.1 | 50.4 | 42.7 |
| ReplaceMe | 44.7 | 53.6 | 24.6 | 52.2 | 66.3 | 57.5 | 50.8 | 150.7 | 36.8 | 23.3 | 40.4 | 55.6 | 51.3 | 41.5 |
| SLEB + R | 50.5 | 61.4 | 32.1 | 52.8 | 70.6 | 56.7 | 54.7 | 69.1 | 42.3 | 25.4 | 42.8 | 61.5 | 51.7 | 44.7 |
| RFS (ours) | 38.2 | 46.8 | 24.1 | 49.0 | 66.9 | 53.6 | 48.1 | 97.6 | 34.9 | 20.5 | 38.4 | 56.1 | 53.1 | 40.6 |
| RGS (ours) | 35.9 | 52.3 | 27.1 | 51.8 | 69.0 | 53.0 | 50.6 | 51.1 | 42.7 | 22.5 | 44.3 | 61.5 | 52.3 | 44.7 |
| RGS-T (ours) | 68.1 | 69.5 | 44.2 | 55.1 | 69.9 | 61.0 | 59.9 | 143.7 | 59.4 | 34.1 | 44.4 | 65.2 | 53.1 | 51.2 |
| Llama-3.1-8B-Instruct, 32 layers: 8 and 12 removed | ||||||||||||||
| Full model | 13.7 | 79.7 | 55.6 | 71.0 | 82.5 | 73.2 | 72.4 | 13.7 | 79.7 | 55.6 | 71.0 | 82.5 | 73.2 | 72.4 |
| ShortGPT + R | 39.4 | 59.3 | 40.3 | 56.5 | 70.4 | 71.1 | 59.5 | 94.0 | 38.2 | 31.6 | 42.0 | 58.1 | 64.1 | 46.8 |
| ReplaceMe | 40.4 | 56.3 | 37.2 | 54.4 | 68.2 | 69.7 | 57.2 | 94.0 | 38.2 | 31.6 | 42.0 | 58.1 | 64.1 | 46.8 |
| SLEB + R | 23.6 | 61.3 | 34.0 | 58.7 | 72.6 | 54.5 | 56.2 | 37.8 | 46.6 | 26.3 | 48.5 | 66.0 | 54.6 | 48.4 |
| RFS (ours) | 24.3 | 52.4 | 27.8 | 54.3 | 69.4 | 51.9 | 51.2 | 95.9 | 32.7 | 23.7 | 37.1 | 54.0 | 49.7 | 39.4 |
| RGS (ours) | 22.6 | 59.9 | 31.3 | 57.7 | 73.5 | 54.4 | 55.4 | 34.4 | 48.4 | 25.6 | 48.7 | 67.1 | 55.2 | 49.0 |
| RGS-T (ours) | 34.6 | 68.1 | 41.1 | 56.1 | 73.1 | 67.6 | 61.2 | 54.0 | 52.3 | 32.0 | 47.5 | 65.3 | 62.6 | 51.9 |
| Qwen3-14B, 40 layers: 10 and 15 removed | ||||||||||||||
| Full model | 15.0 | 83.2 | 60.8 | 67.9 | 80.3 | 73.0 | 73.0 | 15.0 | 83.2 | 60.8 | 67.9 | 80.3 | 73.0 | 73.0 |
| ShortGPT + R | 48.2 | 46.8 | 32.0 | 46.0 | 60.2 | 61.1 | 49.2 | 266.4 | 30.1 | 23.6 | 30.5 | 50.7 | 53.7 | 37.7 |
| ReplaceMe | 41.9 | 55.0 | 30.6 | 51.3 | 66.6 | 53.7 | 51.4 | 43.3 | 42.5 | 26.6 | 43.1 | 59.1 | 51.2 | 44.5 |
| SLEB + R | 27.2 | 48.6 | 28.1 | 49.7 | 65.7 | 53.6 | 49.1 | 43.1 | 37.9 | 24.2 | 43.8 | 57.4 | 51.7 | 43.0 |
| RFS (ours) | 21.9 | 54.8 | 32.1 | 52.4 | 71.4 | 54.2 | 53.0 | 34.0 | 46.3 | 24.8 | 44.6 | 62.0 | 51.4 | 45.8 |
| RGS (ours) | 21.9 | 62.9 | 31.5 | 54.1 | 71.0 | 52.5 | 54.4 | 34.1 | 43.2 | 23.9 | 46.8 | 60.2 | 51.6 | 45.1 |
| RGS-T (ours) | 37.9 | 68.5 | 44.4 | 53.4 | 71.2 | 60.1 | 59.5 | 67.7 | 49.2 | 30.0 | 43.5 | 62.4 | 54.5 | 47.9 |
| Qwen3.8-27B, hybrid attention, 64 layers: 16 and 24 removed | ||||||||||||||
| Full model | 10.1 | 74.4 | 58.5 | 74.1 | 81.9 | 75.3 | 72.8 | 10.1 | 74.4 | 58.5 | 74.1 | 81.9 | 75.3 | 72.8 |
| ShortGPT + R | 34.8 | 33.6 | 23.1 | 41.3 | 53.3 | 51.5 | 40.6 | 42.4 | 32.3 | 21.8 | 38.1 | 52.4 | 48.5 | 38.6 |
| ReplaceMe | 28.7 | 33.8 | 22.0 | 42.5 | 58.3 | 54.4 | 42.2 | 36.9 | 36.4 | 24.5 | 42.5 | 55.7 | 51.9 | 42.2 |
| SLEB + R | 13.2 | 74.8 | 49.6 | 62.6 | 77.9 | 63.8 | 65.7 | 17.1 | 65.8 | 37.7 | 56.1 | 71.3 | 56.0 | 57.4 |
| RFS (ours) | 13.4 | 77.4 | 48.4 | 63.3 | 77.4 | 65.3 | 66.4 | 15.7 | 58.1 | 34.1 | 55.3 | 71.2 | 58.8 | 55.5 |
| RGS (ours) | 13.3 | 73.8 | 49.7 | 63.0 | 77.3 | 60.6 | 64.9 | 16.0 | 66.9 | 37.2 | 55.6 | 74.2 | 55.5 | 57.9 |
Table 13. Per-task results behind Table 3: WikiText-2 perplexity (PPL) and zero-shot accuracy (%) on ARC-Easy, ARC-Challenge, HellaSwag (HS), PIQA and WinoGrande (WG), 1,000 items each. Within each model, the best pruned result is highlighted and the second best underlined.
MMLU. The selection objective decides which layers are kept. 5-shot MMLU (Hendrycks et al. [17]) depends on middle layers that have the smallest effect on WikiText perplexity: on Llama and Qwen3-14B, the WikiText subsets of RGS and SLEB remove them and score 24% to 28%, and the deep blocks of ShortGPT keep 67.1% and 68.4% at 25% removed, as in Gromov et al. [15] (Table 12). With an MMLU objective on validation questions, RGS removes deep layers instead and is the most accurate method at 7 of 8 settings, with 53.3% against at most 26.7% on Qwen3-14B at 15 layers (Section E). The selection objective therefore controls which capabilities are preserved after pruning.
| Model (full) | Removed | ReplaceMe | ShortGPT + R | SLEB + R | WikiText | Tasks | MMLU |
| Qwen3-1.7B (63.5) | 7 | 24.0 | 36.1 | 24.7 | 25.1 | 26.7 | 31.3 |
| 10 | 24.1 | 23.8 | 24.8 | 24.0 | 23.8 | 25.9 | |
| Gemma-3-4B (60.6) | 8 | 24.1 | 24.1 | 27.5 | 25.4 | 36.3 | 53.0 |
| 13 | 23.6 | 23.8 | 23.7 | 25.6 | 24.7 | 27.2 | |
| Llama-3.1-8B (68.4) | 8 | 66.3 | 67.1 | 28.4 | 27.0 | 48.4 | 67.6 |
| 12 | 58.4 | 58.4 | 23.5 | 24.0 | 31.6 | 62.3 | |
| Qwen3-14B (80.8) | 10 | 23.4 | 68.4 | 25.9 | 24.7 | 37.5 | 75.1 |
| 15 | 25.1 | 24.6 | 26.0 | 24.0 | 26.7 | 53.3 | |
| Qwen3.8-27B (83.5) | 16 | 24.1 | – | 54.8 | 52.4 | 57.6 | – |
| 24 | 24.7 | – | 30.2 | 26.0 | 31.2 | – |
Table 12. 5-shot MMLU accuracy (%) on 50 test questions per subject (2,850 questions) of the subsets in Table 3, and of RGS with an MMLU objective: the crossentropy of the correct letter on 5 validation questions per subject; the other methods select without MMLU data. Chance is 25%. All repairs are fitted on WikiText-2. –: not run.
The repair Fisher score
Against the exact repaired loss increase of every block of non-critical groups, the lowest RFS block is within 5% of the best at 19 of 22 block lengths; ReplaceMe’s cosine distance is within 5% at 2. The Spearman correlation is 0.53 to 0.86 for the score and −0.21 to 0.36 for the cosine distance (Figure 5). In Table 3 the score’s block, chosen without evaluating a pruned model, is within 3% of RGS at 4 of 8 settings and beats ReplaceMe at

Figure 5. Ranking blocks under repair without search. (a) Spearman correlation of each score with the exact repaired loss increase of every block of a given length, averaged over lengths (error bars: one standard deviation over lengths). (b) Share of block lengths at which the lowest-scoring block is within 5% of the best block.
7 of 8. At the largest budgets (Gemma at 13 of 34, Llama at 12 of 32), RGS adds layers outside the block and reaches one half to one third of the score’s perplexity, so at heavy budgets the search justifies its cost. The Fisher score thus provides a strong default when a single block is sufficient, and the greedy search extends the method to heavier budgets.
ReplaceMe’s default calibration
ReplaceMe by default fits its maps on SlimOrca conversations (Lian et al. [23]) without a ridge. With this fit, RGS has 16% to 73% lower perplexity on Llama and Gemma (Table 6). With a SlimOrca objective, RGS differs from ReplaceMe only in
| Model | ReplaceMe | RGS | |
| Gemma-3-4B | 8 | 55.9 / 52.4 | 47.0 / 53.6 |
| 13 | 243.7 / 41.6 | 70.5 / 47.7 | |
| Llama-3.1-8B | 8 | 63.7 / 60.2 | 26.1 / 56.8 |
| 12 | 163.1 / 48.8 | 44.4 / 51.0 |
Table 6. ReplaceMe’s calibration (SlimOrca conversations, unregularised least squares) for every method. Perplexity / average accuracy (%).
which layers it removes; its perplexity is lower at 7 of 8 settings and its accuracy at 6 of 8 (Table 7). The advantage of selection under repair therefore holds across calibration data and fitting procedures.
| Model | ReplaceMe | RGS, SlimOrca | |
| Qwen3-1.7B | 7 | 50.5 / 45.3 | 50.9 / 47.2 |
| 10 | 124.9 / 42.1 | 97.8 / 41.6 | |
| Gemma-3-4B | 8 | 55.9 / 52.4 | 54.7 / 54.8 |
| 13 | 243.7 / 41.6 | 93.6 / 47.5 | |
| Llama-3.1-8B | 8 | 63.7 / 60.2 | 28.3 / 57.7 |
| 12 | 163.1 / 48.8 | 49.3 / 50.7 | |
| Qwen3-14B | 10 | 74.2 / 48.2 | 33.5 / 57.3 |
| 15 | 87.4 / 44.2 | 54.7 / 45.7 |
Table 7. ReplaceMe’s calibration with a SlimOrca objective for RGS, disjoint from the fit data; selection and fit use conversation text only. Perplexity / average accuracy (%).
Recovery fine-tuning and a second corpus
RGS has lower C4 perplexity (Raffel et al., [32]) than ReplaceMe at both budgets on all four models, even though it selects on WikiText. After 200 LoRA steps (Hu et al., [18]) on C4 with three seeds, RGS keeps lower mean perplexity at 7 of 8 settings (Figure 6), by 16% to 30% on Llama, 2.5% to 19% on Gemma and 1% to 2% on Qwen3-1.7B. The benefit of better layer selection therefore persists after recovery fine-tuning.

Figure 6. C4 perplexity (a) after repair and (b) after 200 LoRA steps on C4 (mean of three seeds; error bars, one standard deviation). Labels give the change of RGS against ReplaceMe: recovery shrinks the gap, and RGS stays lower at 7 of 8 settings.
Ablations
Table 4 removes one component of RGS at a time on the 8 settings. The repair inside the search matters most: SLEB, which searches without it, has 19% higher perplexity on average (up to 41%). One-shot scoring under repair is 21% worse and ReplaceMe’s criterion 73% worse. Removing the layers that RGS selects without the repair raises perplexity by a factor of four, so the selected subset depends on the repair.
| Variant | PPL / RGS | Worst | Avg | Evals |
| RGS | 1.00 | 1.00 | 48.1 | 303 |
| + shift refinement | 0.995 | 1.00 | 48.4 | 303 |
| − repair in the search (SLEB + R) | 1.19 | 1.41 | 48.0 | 303 |
| − search (one-shot + R) | 1.21 | 1.73 | 46.5 | 34 |
| − search, block criterion (ReplaceMe) | 1.73 | 2.95 | 47.3 | 0 |
| − repair at deployment | 3.98 | 14.9 | – | 303 |
Table 4. Ablation of RGS on the 8 settings of the four enumerated families. PPL / RGS: geometric mean of the perplexity ratio to RGS; Worst: the largest ratio; Avg: mean five-task accuracy (%); Evals: mean number of pruned models evaluated during selection.
Cost
The maps fold into existing weights, so at a given budget every method in Table 3 gives a model of the same size and speed. Removing 25% of the layers gives 1.3× speedup and removing 37.5% gives 1.55×; at 24 of 64 layers, Qwen3.8-27B shrinks from 53.8 to 35.6 GB and fits on one 48 GB GPU (Table 11). RGS evaluates as many pruned models as SLEB and RFS needs one forward and one backward pass (Section E).
| Model | Removed | Weights (GB) | KV (KB/tok.) | Speedup | PPL | |||
| Prefill | Dec. 1 | Dec. 16 | ||||||
| Qwen3-1.7B | 7/28 | 1.32 | 1.34 | 1.32 | 37.72 | |||
| 10/28 | 1.55 | 1.53 | 1.54 | 54.25 | ||||
| Llama-3.1-8B | 8/32 | 1.32 | 1.30 | 1.31 | 22.65 | |||
| 12/32 | 1.58 | 1.54 | 1.55 | 34.43 | ||||
| Qwen3-14B | 10/40 | 1.30 | 1.31 | 1.31 | 21.96 | |||
| 15/40 | 1.57 | 1.55 | 1.56 | 34.26 | ||||
| Qwen3.8-27B | 16/64 | 1.29 | 1.31 | 1.29 | 13.26 | |||
| 24/64 | 1.55 | 1.58 | 1.55 | 16.03 |
Table 11. Inference cost of the folded RGS models in bfloat16 on RTX A6000 GPUs (two for Qwen3.8-27B): weights, KV cache per token, and speedup over the full model for prefill of 2,048 tokens and for decoding 256 tokens at batch 1 and 16. PPL is the WikiText-2 perplexity of the folded model; it matches the model with the maps applied as separate operations to within 0.05. Gemma-3 normalises the MLP output before the residual addition, so its maps cannot be folded and it is not listed. Only the full-attention layers of Qwen3.8-27B keep a KV cache.
Conclusion
The exact optimum of depth pruning depends on the repair: local super-additivity makes a contiguous block close to the worst choice without repair and close to the best with it. Evaluating each choice on the repaired model that will be deployed lowers perplexity by 20% to 66% relative to ReplaceMe at the cost of SLEB, and a second-order expansion of the repaired loss gives a score computed from the intact model.
The core finding applies whenever the repair alters the cost landscape of the removed layers. Future work can test whether it holds for low-rank or shifted repairs. The repair Fisher score shows that repair quality can be measured from the intact model alone, which opens a path to joint optimisation of layer selection and the repair objective.
Limitations
Exact optima are available only at two-layer granularity up to 6 removed groups and at single-layer granularity up to 3; at heavier budgets we compare methods with each other. The structural results concern perplexity: at two-layer granularity the perplexity optimum is the most accurate subset, or within one standard error of it, in 10 of 14 cases, and in the other four a different subset is 2 to 7 points more accurate. We test one repair, ReplaceMe’s least-squares map; two schemes that refit later maps on the already repaired network did not improve on it, and newer repairs ([4, 40] (Chen et al. 2026)) are not tested. We do not compare with BlockPruner or SubFit. Recovery used one length, 200 LoRA steps. ReplaceMe’s example configuration fits on conversations of up to 1,024 tokens with the model in 4 bits; we fit on blocks of 256 tokens with the model in bfloat16. We evaluate only English text and multiple-choice tasks; multilingual text and open-ended generation remain untested.
References
Proof of Proposition 1
Fix a position and write and . Then
At the divergence is zero. Its gradient is , because and the probabilities sum to one for every . With and , its Hessian is
Summing over positions and expanding to second order gives Equation (3). The remainder is when the downstream network is three times differentiable in the residual stream, which holds for transformer layers with smooth activations and normalisation away from zero norm.
Both approximations in § 6.2 are standard in the pruning literature. The empirical Fisher replaces the model’s predictions by the observed tokens; the two agree when the model is calibrated on the calibration text. The block-diagonal rank-one form drops the curvature between positions and, within a position, in directions other than . We validate the resulting score against the exact repaired loss increases in § 7.3.
Algorithms
We always keep the first layer: the map of a run is folded into the kept layer before the run, so a run that starts at the first layer cannot be repaired. In every enumerated setting the first group is also critical.
Algorithm 1 Greedy selection under repair (RGS)
Require: layers , budget , objective , activations of the intact model
for do
for each kept layer do
fit by (1) for each run of ▷ cached
end for
end for
return , with its maps folded into the down-projections
Mechanism of local super-additivity
Table Table 5 lists the explanations we tested. Only functional similarity has partial support: given the distance, the linear CKA of the updates of two groups also predicts their pairwise term on Qwen3-1.7B and Llama, weakly on Qwen3-14B, and not on Gemma.
| Explanation | Test | Result |
| Distribution shift of the residual stream | replace removed groups by their mean update | ruled out: pairwise terms remain (+0.55 against +0.38 for neighbours) |
| Later layers restore the update | share of the removed update that later layers restore | ruled out: Spearman +0.17 with the pairwise terms, against +0.72 for distance |
| Convexity of the loss | second-order split of each pairwise term | ruled out: curvature term +0.24 at long distance, where the pairwise term is zero |
| Logit effects add | exact split into loss and network parts | ruled out: joint logit changes differ from the sum by 50% to 70% |
| Nearby layers compute similar functions | partial Spearman of linear CKA and the pairwise term, given distance | Qwen3-1.7B +0.37 ( = 0.006), Llama +0.36 (0.001), Qwen3-14B +0.11 (0.09), Gemma −0.06 (0.68) |
Table 5. Tests of explanations for local super-additivity.
Search without repair
Over the 20 exactly enumerated budgets with at least two removed groups, SLEB finds the optimum in 13 (Figure 7). In 3 of its 7 misses its subset is one shift of one removed group away from the optimum: the optimum keeps a group between two removed runs that greedy removal took out early. A refinement (iterative removal over non-critical groups followed by shift moves, at SLEB’s cost), designed on Qwen3-1.7B, finds the optimum in 16 budgets in every seed and in 2 more in 6% and 68% of seeds. It improves on SLEB on Gemma; on Qwen3-14B at 3 and 4 removed groups SLEB’s subset is already a local optimum under single swaps.

Figure 7. Calibration perplexity of each search method relative to the exact optimum without repair, at every exactly enumerated budget; randomised methods are averaged over 100 seeds.
Additional results
MMLU. Table 12 gives 5-shot MMLU accuracy for every subset. The MMLU objective uses 5 questions per subject from the validation split, 285 in total; the test questions and the dev split, which supplies the 5-shot examples, are disjoint from them. With it RGS is the most accurate method at 7 of 8 settings, by 0.5 to 26.6 points. The exception is Qwen3-1.7B at 7 layers, where ShortGPT reaches 36.1% and RGS 31.3%. On Qwen3-1.7B at 10 and Gemma at 13 layers all methods are within 3 points of chance. Each objective favours the metric that it optimises: the MMLU subsets have 1.7 to 4.9 times the WikiText perplexity of the WikiText subsets, and on Qwen3-14B a five-task accuracy 3 to 10 points lower. At 25% removed, the WikiText subsets of RGS and SLEB on Llama and Qwen3-14B put 0.80 to 0.93 of the probability on the four answer letters, close to the full model’s 0.999, and each answers its most frequent letter for at most 56% of the questions, so they follow the answer format. ReplaceMe’s block on Qwen3-14B at 10 layers puts 0.06 on the letters and answers A for 80% of the questions. On Qwen3.8-27B the task objective gives the highest MMLU accuracy at both budgets: 57.6% at 16 layers against 54.8% for SLEB and 24.1% for ReplaceMe, and 31.2% at 24 layers against 30.2% for SLEB. The gain of the task objective thus carries over from the zero-shot tasks to knowledge questions on the largest model.
Selection time. On Llama-3.1-8B at 12 of 32 layers on one RTX A6000, the selection takes 384 seconds for RGS (319 pruned models), 343 for SLEB with repair, 487 for LaCo, 56 for Prune&Comp, 33 for RFS, 8 for ReplaceMe and 1 for ShortGPT. Each repaired method also needs 29 seconds to collect the activations for the maps.
Calibration size. With 16 calibration sequences the optimum on the calibration sequences is also the optimum on the test sequences in 13 of 17 enumerated settings. Resampling calibration sets from a pool of 64 shows that the disagreement is sampling noise: agreement rises to 100% at 64 sequences on Qwen3-1.7B and Gemma-3-4B.
Repair fitting. Two schemes that fit later maps on the already repaired network did not improve on ReplaceMe’s map. Fitting towards the activations of the intact network made multi-run subsets 1% to 16% worse on Qwen3-1.7B and Llama; fitting towards the output of the removed run, computed on the inputs of the repaired network, was about 5% better on Qwen3-1.7B and within 1.5% either way on Llama. These differences are small compared with the differences between selection methods in Table 3, so the gains of RGS come from the choice of layers.
Per-task results
Per-task accuracy shows where the WikiText objective and task accuracy disagree. At 25% removed, ShortGPT’s and ReplaceMe’s deep blocks keep WinoGrande on Llama (71.1 and 69.7 against 54.4 for RGS), while RGS with the WikiText objective removes middle layers that WinoGrande needs but that have little effect on perplexity. The task objective recovers this: RGS-T reaches 67.6 on WinoGrande there, has the best ARC-Easy accuracy on all four families at both budgets, and the best ARC-Challenge accuracy at 6 of these 8 settings.
Implementation details
Models run in bfloat16. Sequences begin with a BOS token when the tokenizer adds one. Repair maps are solved in float32 with the ridge; without a ridge they are solved in float64 with a pseudo-inverse, because Gemma’s MLP outputs on SlimOrca are rank-deficient. For Gemma 3 the map acts on the output of the post-feedforward normalisation, which cannot be folded into a weight, so the repaired Gemma model has one extra
matrix per run. Qwen3.8-27B runs split over two GPUs. During the study we found and fixed two faults, each recorded with its effect in the repository: removed or moved Gemma layers received the wrong attention type, and unregularised maps were first solved by a routine that assumes full rank. Every number in this paper comes from runs after the fixes. All data behind the tables and figures is exported to CSV files in the supplementary material.
| Model | ReplaceMe | ReplaceMe, task data | RGS-T | |
| Qwen3-1.7B | 7 | 45.6 | 43.8 | 47.1 |
| 10 | 40.9 | 40.9 | 41.7 | |
| Gemma-3-4B | 8 | 50.8 | 45.8 | 59.9 |
| 13 | 41.5 | 40.6 | 51.2 | |
| Llama-3.1-8B | 8 | 57.2 | 58.9 | 61.2 |
| 12 | 46.8 | 46.8 | 51.9 | |
| Qwen3-14B | 10 | 51.4 | 51.4 | 59.5 |
| 15 | 44.5 | 38.8 | 47.9 |
Table 8. Average accuracy (%) with the task objective, against ReplaceMe’s criterion computed on the hidden states of the same task training texts.
| Model | C4 PPL | WikiText PPL | Avg | |
| Qwen3-1.7B | 7 | 25.5 / 25.0 | 29.3 / 28.8 | 52.0 / 50.8 |
| 10 | 31.0 / 30.7 | 34.4 / 34.2 | 47.1 / 47.3 | |
| Gemma-3-4B | 8 | 20.1 / 19.6 | 27.4 / 25.6 | 56.1 / 56.9 |
| 13 | 31.2 / 25.5 | 43.9 / 35.4 | 48.5 / 51.2 | |
| Llama-3.1-8B | 8 | 21.9 / 18.3 | 29.2 / 21.3 | 62.7 / 58.6 |
| 12 | 30.7 / 22.9 | 39.4 / 27.4 | 56.2 / 54.4 | |
| Qwen3-14B | 10 | 18.8 / 17.3 | 19.9 / 17.3 | 59.6 / 60.2 |
| 15 | 22.2 / 22.7 | 22.9 / 23.6 | 51.7 / 54.1 |
Table 10. After 200 LoRA steps on C4, mean of three seeds: ReplaceMe / RGS. The spread between seeds is 0.01 to 0.3 in C4 perplexity, 0.06 to 0.66 in WikiText perplexity and at most 0.5 points in accuracy.
References
- [1]Ali Bahri, Hang Li, Hongliang Li, and Zhitang Chen. 2026. SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs. CoRR, abs/2608.25068.arxiv.org/abs/2608.25068
- [2]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020., pages 7432–7439. AAAI Press.arxiv.org/abs/1911.11641
- [3]Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. 2025a. Streamlining Redundant Layers to Compress Large Language Models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.arxiv.org/abs/2403.19135
- [4]Xinrui Chen, Haoli Bai, Tao Yuan, Ruikang Liu, Kang Zhao, Xianzhi Yu, Lu Hou, Tian Guan, Yonghong He, and Chun Yuan. 2025b. A Simple Linear Patch Revives Layer-Pruned Large Language Models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.arxiv.org/abs/2505.24680
- [7]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2924–2936. Association for Computational Linguistics.arxiv.org/abs/1905.10044
- [8]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. CoRR, abs/1803.05457.arxiv.org/abs/1803.05457
- [9]Elia Cunegatti, Marcus Vukovjevic, Erik Nielsen, and Giovanni Iacca. 2026. From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression. CoRR, abs/2606.02559.arxiv.org/abs/2606.02559
- [10]Xuan Ding, Pengyu Tong, Ranjie Duan, Yunjian Zhang, Rui Sun, and Yao Zhu. 2026. Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models. In The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc.
- [11]Elias Frantar and Dan Alistarh. 2022. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.arxiv.org/abs/2208.11580
- [12]Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA., volume 202 of Proceedings of Machine Learning Research, pages 10323–10337. PMLR.arxiv.org/abs/2301.00774
- [13]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.arxiv.org/abs/2210.17323
- [15]Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. 2025. The Unreasonable Ineffectiveness of the Deeper Layers. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.arxiv.org/abs/2403.17887
- [16]Babak Hassibi and David G. Stork. 1993. Second Order Derivatives for Network Pruning: Optimal Brain Surgeon. In Advances in Neural Information Processing Systems 5, [NIPS Conference, Denver, Colorado, USA, November 30 - December 3, 1992], pages 164–171. Morgan Kaufmann.
- [17]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- [18]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yunzh i Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.arxiv.org/abs/2106.09685
- [19]Bo-Kyeong Kim, Geon min Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. 2024. Shortened LLaMA: A Simple Depth Pruning for Large Language Models. CoRR, abs/2402.02834.
- [20]Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim, Youngjin Heo, Suin Cho, Seong hun Kim, Woosang Lim, and Gaeul Kwon. 2026. Rethinking Layer Redundancy in Large Language Models: Calibration Objectives and Search for Depth Pruning. CoRR, abs/2604.24938. Cited version: arXiv:2604.24938v1. DBLP metadata corresponds to this version.
- [21]Yann LeCun, John S. Denker, and Sara A. Solla. 1990. Optimal Brain Damage. In Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989], pages 598–605. Morgan Kaufmann.
- [22]Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.arxiv.org/abs/2102.05426
- [23]Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanviet Yong, and Teknium. 2023. SlimOrca: An Open Dataset of GPT-4 Augmented FLAN Reasoning Traces, with Verification.
- [24]Kainan Liu, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, and Jing Xiao. 2025. GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 26333–26348. Association for Computational Linguistics.arxiv.org/abs/2501.00339
- [25]Llama Team. 2024. The Llama 3 Herd of Models. CoRR, abs/2407.21783.
- [26]Yao Lu, Yuqi Li, Wenbin Xie, Shanqing Yu, Qi Xuan, Zhaowei Zhu, and Shiping Wen. 2025. The Structural Scalpel: Automated Contiguous Layer Pruning for Large Language Models. CoRR, abs/2510.23652.arxiv.org/abs/2510.23652
- [27]Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2025. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 20192–20204. Association for Computational Linguistics.
- [28]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.arxiv.org/abs/1609.07843
- [29]Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance Estimation for Neural Network Pruning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 11264–11272. Computer Vision Foundation / IEEE.arxiv.org/abs/1906.10771
- [30]B. Moore. 1981. Principal component analysis in linear systems: Controllability, observability, and model reduction. IEEE Transactions on Automatic Control, 26(1):17–32.DOI
- [32]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21:140:1–140:67.arxiv.org/abs/1910.10683
- [33]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732–8740. AAAI Press. Dmitriy Shopkhoev, Ammar Ali, Magauiya Zhussip, Valentin Malykh, Stamatios Lefkimmatis, Nikos Komodakis, and Sergey Zagoruyko. 2025. ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.
- [34]Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas M. Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov. 2024. A deeper look at depth pruning of LLMs. CoRR, abs/2407.16286.
- [35]Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, and Dan Alistarh. 2025. EvoPress: Accurate Dynamic Model Compression via Evolutionary Search. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net.arxiv.org/abs/2410.14649
- [36]Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. 2024. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 46136–46155. PMLR / OpenReview.net.arxiv.org/abs/2402.09025
- [37]Lintang Sutawika, Leo Gao, Hailey Schoelkopf, Stella Biderman, Jonathan Tow, Baber Abbasi, ben fattori, Charles Lovering, farzanehkhaee70, Jason Phang, Anish Thite, Fazz, Aflah, Niklas Muennighoff, Thomas Wang, sdtblck, nopperl, gakada, tttyuntian, and 11 others. 2023. EleutherAI/lm-evaluation-harness: Major refactor. Software, version v0.4.0.DOI
- [38]Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. 2018. Faster gaze prediction with dense networks and Fisher pruning. CoRR, abs/1801.05787.arxiv.org/abs/1801.05787
- [39]Yifei Yang, Zouying Cao, and Hai Zhao. 2024. LaCo: Large Language Model Pruning via Layer Collapse. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, volume EMNLP 2024 of Findings of ACL, pages 6041–6417. Association for Computational Linguistics.arxiv.org/abs/2402.11187
- [40]Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, and Sunwoo Lee. 2026. Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs. CoRR, abs/2605.15491.arxiv.org/abs/2605.15491
- [41]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics.arxiv.org/abs/1905.07830
- [42]Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. 2025. BlockPruner: Fine-grained Pruning for Large Language Models. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 5065–5080. Association for Computational Linguistics.arxiv.org/abs/2406.10594
