WorksComputer Science
Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization
Abstract
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical. We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight–activation saliency to construct task-conditioned sparse perturbation masks while keeping the pretrained model dense. We analyze support quality through gradient energy capture and relate it to gradient estimation and convergence. Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps. Comparisons with random and magnitude-based sparse perturbations further highlight the importance of informative support selection.
Introduction
Fine-tuning large language models (LLMs) requires substantial memory for the activations, gradients, and optimizer states used in backpropagation. Parameter-efficient fine-tuning [15, 20], quantization [7], and memory-efficient optimizers [38, 39] reduce different parts of this cost. Zeroth-order (ZO) optimization offers another approach by estimating update directions through forward evaluations, as demonstrated by MeZO [23]. Its reduced training-memory requirements are particularly appealing for on-device adaptation, where models must be personalized under limited device resources [14].
Despite this memory advantage, ZO optimization remains challenging in the high-dimensional parameter spaces of LLMs. The variance of random-direction gradient estimates grows with the number of perturbed coordinates, making it difficult to obtain informative updates. Sparse ZO methods address this issue by restricting perturbations to a subset of parameters. However, reducing the number of coordinates is only part of the problem: the selected support must also retain useful information for downstream optimization. Existing methods select small-magnitude weights [21] or transfer sensitivity information from pre-training [14]. This motivates selecting sparse supports using information available from the downstream task itself.
We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight–activation saliency [29] to select task-conditioned perturbation coordinates. A lightweight forward calibration produces a row-wise sparse mask that is reused during MeZO updates, while the pretrained model remains dense. In this way, downstream activation information guides the allocation of a fixed perturbation budget.
To understand the effect of coordinate selection, we study gradient energy capture, which measures the proportion of gradient energy contained in the selected support. This perspective connects support quality to the alignment of ZO estimates and to convergence. We complement the analysis with comparisons against alternative sparse supports and evaluations across downstream tasks and model scales.
Our main contributions are as follows.
Task-Conditioned Sparse Perturbations. We introduce WIS-MeZO, which selects sparse perturbation coordinates with downstream Wanda saliency while retaining the dense pretrained model and requiring only forward calibration.
Gradient-Energy Analysis. We analyze the relationship between saliency-based support selection and captured gradient energy, and establish theoretical results connecting support quality to ZO estimation and convergence.
Empirical Evaluation. Experiments across 16 benchmarks and LLaMA and Qwen models up to 32B parameters show improved aggregate performance over MeZO. Comparisons with random and magnitude-based sparse perturbations, together with validation-loss curves, further demonstrate the benefit of informed support selection.
Related Work
Zeroth-order optimization estimates update directions from function evaluations instead of backpropagation [28, 16]. Classical approaches include ZO-SGD [11] and ZO-Adam [3], while MeZO [23] enables memory-efficient LLM fine-tuning through seed-replayed perturbations. Subsequent work improves ZO estimation through variance reduction [10], preconditioning [37], additional queries [6], and low-rank or structured perturbations [36, 33, 4]. AGZO [19] also uses downstream activations, constructing low-rank perturbation subspaces from current mini-batches. A complementary approach is to reduce the search dimension by selecting a sparse set of weight coordinates.
Sparse coordinate selection has been studied in ZO optimization [25, 13, 1, 35]. DeepZero [2] exploits finite-difference information, Sparse-MeZO [21] selects small-magnitude weights, and SensZOQ [14] uses sensitivity information from pre-training. Separately, Wanda [29] and SparseGPT [9] study weight importance for pruning, while dynamic sparse training [8, 17] adapts sparse networks during optimization. Building on Wanda’s weight–activation score, WIS-MeZO uses downstream data to select perturbation coordinates while keeping the model dense. We analyze the quality of this support through gradient energy capture.
Preliminaries
We first review the MeZO estimator and the Wanda score, which together form the basis of WIS-MeZO.
Zeroth-Order Optimization
For a linear layer, let denote the weight matrix and let be its gradient on a training mini-batch . MeZO [23] estimates this gradient using a random direction and a perturbation radius ,
The entries of are independent standard Gaussian variables. Each estimate uses one direction and two loss evaluations on the same mini-batch. As , its expectation over approaches . The update is .
The matrix notation describes a single layer with coordinates. For full-model updates, and the corresponding masks denote the collections of parameter blocks being optimized. Their inner products and squared Frobenius norms sum over these blocks. We use
for the total candidate dimension and the number of selected coordinates, respectively. A common row-wise density gives .
Wanda Importance Score
Wanda [29] scores each weight by its magnitude and the activation norm of its input feature. For a weight matrix and calibration activations , the score is
WIS-MeZO uses this score to select the perturbation mask.
Methodology
Building on these two ingredients, WIS-MeZO uses Wanda scores to choose perturbation coordinates before applying MeZO updates on the selected support. We describe the support construction and optimization procedure below.
WIS-Guided Sparse Perturbation
For each layer , let be a binary mask. Given a density , WIS-MeZO selects the largest scores in each row of . The mask records which coordinates receive perturbations and updates; the underlying weight matrix remains dense. Row-wise selection allocates the same perturbation budget to every output unit.
The perturbation direction is , with independent standard Gaussian entries in . For one layer, this gives while restricting the support to coordinates. Across the model, the corresponding support dimension is .
Calibration from Downstream Data
WIS-MeZO constructs the mask from 128 examples of the downstream training data using Wanda scores. This calibration is performed once before optimization; WISD-MeZO periodically refreshes the mask every steps. Appendix § 7.3 examines the effect and cost of calibration size.
The WIS-MeZO Algorithm
In each iteration , we sample a single random seed to generate the perturbation . The gradient estimator uses one sampled direction and two loss evaluations on the same training mini-batch,
Algorithm 1 WIS-MeZO
Input: Initial weights , step sizes , perturbation scale , mask update frequency , density ratio , calibration dataset
for step do
if then
Fetch activations {Collect calibration activations}
{Update mask using Wanda scores}
end if
Sample training batch and random seed
{Move to the negative perturbation}
{Recover parameters}
Reset the random number generator with seed and regenerate
{ZO gradient estimate}
{Update the parameters}
end for
Here, denotes element-wise multiplication, and the update is . Seed replay regenerates perturbations for in-place updates without storing full-model gradients. Algorithm 1 summarizes WIS-MeZO, with supplementary pseudocode in Appendix § 12.
With in-place perturbations and seed replay, the main additional storage is the binary mask. A bit-packed representation reduces its cost from 1 byte to 1 bit per candidate parameter, requiring less than 4GB for the 32B model. Together with one-time calibration, this compact storage makes WIS-MeZO a promising approach for memory-constrained on-device adaptation. § 7.1 reports peak GPU memory and throughput.
Theoretical Analysis
Sparse perturbations reduce the search dimension, but different supports retain different amounts of gradient information. We analyze this trade-off through gradient energy capture, first relating Wanda scores to gradient energy and then studying estimator alignment and convergence.
Energy Analysis
For the analysis, let be the per-example gradient, with denoting the backpropagated signal and an input activation. Write and for the population and empirical activation second moments.
Assumption 5.1 (Bounded Input–Error Dependence). There exist constants such that
The constants and describe the dependence between squared inputs and squared error signals. Independence gives .
Assumption 5.2 (Calibration Concentration). At the parameter value used for calibration, the activation observations are independent draws from the downstream distribution. For every input feature,
where is the sub-Gaussian norm.
Under this assumption, gives simultaneous relative accuracy over the input features with probability at least , for .
Let be the WIS-selected support in row , and let contain the largest expected gradient energies among all coordinates in that row. Set . For rows with nonzero gradient energy, suppose
Define
These factors account for input–error dependence, weight variation within the compared supports, and calibration error.
Lemma 5.3 (Gradient-Saliency Proxy Bound). Under Assumptions 5.1 and 5.2, if , then with probability at least ,
simultaneously for , where .
The lemma relates the Wanda ranking to expected gradient energy.
For a fixed support and nonzero expected gradient energy, define its energy capture ratio by
where
Rows with zero gradient energy contribute zero to the sum. Let be the capture ratio of the row-wise oracle supports .
Theorem 5.4 (Energy Capture Relative to the Oracle). Under the conditions of Lemma 5.3, with probability at least ,
A uniform row-wise random mask of density has expected capture ratio at the same parameter value. Thus, is a sufficient condition for WIS to exceed this random-mask reference.
The result separates gradient-energy concentration, measured by , from the accuracy of the saliency ranking, measured by . Calibration affects the latter through .
Figure 2 compares support rules on common mini-batch gradient snapshots from an auxiliary AdamW run. All masks are constructed at step 0 and remain fixed. At , the plotted WIS supports retain over 40% of the gradient energy, compared with the 5% random-mask expectation. WIS also captures more energy than W-small and W-large in these diagnostics. W-small exceeds random masking, while W-large falls below it, so the observed energy ordering is not determined by increasing weight magnitude. These measurements evaluate how the initial supports align with gradients encountered during the diagnostic run. Results at additional densities appear in Appendix § 11.
Figure 2. Gradient energy capture ratio on LLaMA-3.2-3B at a density ratio of . We compare uniform random masking, the small-magnitude mask used to represent the selection principle of Sparse-MeZO (W-small), the corresponding large-magnitude mask (W-large), and the Wanda-based WIS mask.
We next consider estimator alignment at a fixed parameter value. For the mini-batch gradient , define by Eq. equation 1 using the mini-batch distribution. Let be the total number of selected coordinates and , with standard Gaussian entries in .
Lemma 5.5 (Moment-Normalized Alignment). For the directional-derivative estimator with ,
At the same gradient distribution, averaging over uniform row-wise masks replaces by .
The estimator is the limit of the central-difference estimator. The lemma isolates the contribution of support quality at a fixed dimension. The next result includes the finite perturbation radius used by the algorithm.
Convergence Analysis
We analyze the fixed support produced at calibration. In this subsection, , , and denote the full collections of parameter blocks, and squared norms sum over all blocks.
Assumption 5.6 (-Smooth Mini-Batch Losses). Each mini-batch loss is differentiable and -smooth. The population objective is bounded below by .
Assumption 5.7 (Unbiased Gradients with Bounded Variance). Conditioned on the optimization history, satisfies
The perturbation is independent of the current mini-batch.
Theorem 5.8 (Finite-Difference Convergence). Suppose Assumptions 5.6 and 5.7 hold. For a fixed mask, let , , , and
Assume a deterministic constant satisfies
almost surely for . With Gaussian perturbations, radius , and , the central-difference iterates satisfy
The convergence bound depends on the selected dimension, the gradient energy retained during training, and the finite-difference radius. The result also highlights the role of support quality in sparse ZO optimization.
Experiments
We now examine how task-conditioned support selection affects downstream fine-tuning. We evaluate five GLUE tasks [30], SQuAD, and five SuperGLUE tasks [31], followed by commonsense and question-answering benchmarks across larger model scales. Additional efficiency and sensitivity results appear in the appendix.
Fine-Tuning on GLUE, SQuAD, and SuperGLUE
We evaluate WIS-MeZO and its dynamic variant WISD-MeZO on LLaMA-3.2-3B [12] and Qwen2.5-3B [32]. All ZO methods use a batch size of 16 and train for 20,000 steps. Hyperparameters are reported in Appendix § 10.
The first-order baselines are full-parameter fine-tuning (FT) with AdamW [18, 22] and LoRA [15]. The ZO baselines are MeZO [23], Sparse-MeZO [21], which selects small-magnitude weights, and Random-MeZO, which uses random sparse supports. These comparisons evaluate the effect of coordinate selection within sparse ZO fine-tuning.
Table 1 and Table 2 report the task results. Figure 1 compares validation loss against optimization steps over five random seeds.
Figure 1. Validation loss versus training steps for fine-tuning LLaMA3.2-3B, averaged over five random seeds. WIS-MeZO reaches comparable validation-loss levels in fewer optimization steps.
Table 1: Results on five GLUE tasks and SQuAD (five random seeds). We report F1 scores for QQP, MRPC, and SQuAD, and accuracy (%) for all other tasks (↑). FT and LoRA denote full parameter fine-tuning and Low-Rank Adaptation, respectively, both utilizing the AdamW optimizer.
| Models | Methods | MNLI | RTE | QQP | MRPC | SST-2 | SQuAD | Avg. |
| LLaMA3.2-3B | Zero-shot | 38.3 | 49.4 | 53.7 | 78.3 | 58.8 | 55.0 | 55.6 |
| FT | 86.5 | |||||||
| LoRA | 86.6 | |||||||
| MeZO | 74.4 | |||||||
| Sparse-MeZO | 77.7 | |||||||
| Random-MeZO | 76.4 | |||||||
| WIS-MeZO | 79.1 | |||||||
| WISD-MeZO | 79.8 | |||||||
| Qwen2.5-3B | Zero-shot | 74.8 | 80.5 | 34.1 | 51.1 | 56.0 | 83.6 | 63.4 |
| FT | 87.5 | |||||||
| LoRA | 87.4 | |||||||
| MeZO | 85.4 | |||||||
| Sparse-MeZO | 86.6 | |||||||
| Random-MeZO | 86.0 | |||||||
| WIS-MeZO | 86.9 | |||||||
| WISD-MeZO | 87.0 |
Table 1. Results on five GLUE tasks and SQuAD (five random seeds). We report F1 scores for QQP, MRPC, and SQuAD, and accuracy (%) for all other tasks (↑). FT and LoRA denote full parameter fine-tuning and Low-Rank Adaptation, respectively, both utilizing the AdamW optimizer.
| Models | Methods | CB | MultiRC | WSC | ReCoRD | BoolQ | Avg. |
| LLaMA3.2-3B | Zero-shot | 44.6 | 47.0 | 36.5 | 79.7 | 61.6 | 53.9 |
| FT | 91.0 | 83.6 | 57.6 | 81.6 | 83.1 | 79.4 | |
| LoRA | 92.8 | 85.2 | 62.5 | 81.4 | 84.3 | 81.2 | |
| MeZO | 71.4 | 72.4 | 61.5 | 81.2 | 77.6 | 72.8 | |
| Sparse-MeZO | 67.8 | 79.7 | 62.5 | 81.4 | 81.3 | 74.5 | |
| Random-MeZO | 75.0 | 74.2 | 61.5 | 81.0 | 78.9 | 74.1 | |
| WIS-MeZO | 82.1 | 76.3 | 63.4 | 82.0 | 82.6 | 77.3 | |
| WISD-MeZO | 82.1 | 76.3 | 64.4 | 81.9 | 83.0 | 77.6 | |
| Qwen2.5-3B | Zero-shot | 30.3 | 65.8 | 63.4 | 72.1 | 70.3 | 60.3 |
| FT | 96.4 | 88.0 | 68.4 | 74.4 | 80.9 | 81.6 | |
| LoRA | 94.6 | 86.9 | 67.4 | 76.0 | 85.1 | 82.0 | |
| MeZO | 96.4 | 83.2 | 66.3 | 73.7 | 80.9 | 80.1 | |
| Sparse-MeZO | 92.8 | 86.1 | 67.3 | 75.6 | 80.1 | 80.3 | |
| Random-MeZO | 96.4 | 85.6 | 65.3 | 74.5 | 81.6 | 80.9 | |
| WIS-MeZO | 96.4 | 86.4 | 68.2 | 74.9 | 82.6 | 81.7 | |
| WISD-MeZO | 98.2 | 86.9 | 71.1 | 75.6 | 82.8 | 82.9 |
Table 2. Experimental results on SuperGLUE. WIS-based methods achieve the strongest aggregate ZO performance on both model families.
WIS-MeZO improves the average score over MeZO, Random-MeZO, and Sparse-MeZO in all four model–benchmark combinations. On LLaMA-3.2-3B, its average scores are 79.1 on the GLUE/SQuAD tasks and 77.3 on SuperGLUE, compared with 77.7 and 74.5 for Sparse-MeZO. The corresponding Qwen2.5-3B scores are 86.9 and 81.7, compared with 86.6 and 80.3. The size of the gain varies across backbones and tasks, with the larger improvements occurring on LLaMA-3.2-3B.
Beyond final task scores, the validation-loss curves in Figure 1 show that WIS-MeZO reaches comparable loss levels in fewer optimization steps than MeZO on RTE, MNLI, and BoolQ. This improvement is consistent with the gradient-energy results in § 5. Runtime and memory measurements are reported in § 7.1.
Periodic recalibration in WISD-MeZO increases the average score in all four model–benchmark combinations. Fixed-mask WIS-MeZO is the primary method, and WISD-MeZO characterizes the effect of refreshing the support during training.
Scaling to Large-Scale LLMs
Having compared support-selection strategies on 3B models, we next examine whether the improvement over MeZO extends to larger models. We evaluate fixed-mask WIS-MeZO on Qwen2.5 models from 3B to 32B parameters using ARC-E, ARC-C [5], HellaSwag [34], WinoGrande [27], and OpenBookQA [24]. Table 3 shows higher average scores at every scale and performance matching or exceeding MeZO on every task. On WinoGrande, WIS-MeZO also improves over the zero-shot model at all four scales.
| Models | Methods | ARC-E | ARC-C | HellaSwag | WinoGrande | OpenBookQA | Avg. |
| Qwen2.5-3B | Zero-shot | 70.0 | 48.4 | 68.0 | 66.3 | 40.8 | 58.7 |
| MeZO | 82.9 | 57.8 | 69.9 | 66.3 | 49.0 | 65.1 | |
| WIS-MeZO | 85.4 | 60.8 | 74.4 | 69.7 | 54.0 | 68.8 | |
| Qwen2.5-7B | Zero-shot | 74.9 | 52.5 | 73.8 | 68.9 | 43.8 | 62.7 |
| MeZO | 83.0 | 59.2 | 74.9 | 66.5 | 51.2 | 66.9 | |
| WIS-MeZO | 85.0 | 61.2 | 79.4 | 70.7 | 53.8 | 70.0 | |
| Qwen2.5-14B | Zero-shot | 75.4 | 53.8 | 75.8 | 70.6 | 43.2 | 63.7 |
| MeZO | 84.0 | 67.2 | 77.2 | 73.8 | 57.2 | 71.8 | |
| WIS-MeZO | 87.8 | 70.9 | 81.5 | 76.6 | 58.8 | 75.1 | |
| Qwen2.5-32B | Zero-shot | 74.5 | 54.5 | 79.0 | 74.3 | 43.6 | 65.1 |
| MeZO | 85.9 | 69.8 | 78.7 | 74.9 | 52.4 | 72.3 | |
| WIS-MeZO | 88.7 | 69.8 | 84.5 | 77.9 | 56.2 | 75.4 |
Table 3. Experimental results for Qwen2.5-3B to 32B. WIS-MeZO consistently improves average performance over MeZO across all model scales.
Additional Experimental Results
Memory and Speed Comparison
We report peak GPU memory, iteration throughput, and total training time in Table 4 and Table 5. Table 6 compares standard and bit-packed masks on Qwen2.5-32B.
| Methods | SST-2 Mem. | SST-2 Iter/s | ReCoRD Mem. | ReCoRD Iter/s | CB Mem. | CB Iter/s |
| FT | 35.5 | 3.37 | 87.4 | 0.92 | 94.5 | 0.75 |
| LoRA | 22.1 | 4.73 | 73.5 | 1.33 | 85.0 | 0.94 |
| MeZO | 9.70 | 3.50 | 21.5 | 1.39 | 45.1 | 0.99 |
| Sparse-MeZO | 16.2 | 2.98 | 27.3 | 1.30 | 51.0 | 0.95 |
| WIS-MeZO | 11.6 | 3.16 | 25.1 | 1.33 | 47.0 | 0.97 |
Table 4. Peak GPU memory (GB, batch size=16) and iteration speed (iteration/s) for fine-tuning Qwen2.5-3B.
| Methods | SST-2 | ReCoRD | CB |
| MeZO | 1.59 | 4.00 | 5.61 |
| Sparse-MeZO | 1.86 | 4.27 | 5.85 |
| WIS-MeZO | 1.76 | 4.18 | 5.73 |
Table 5. End-to-end wall-clock time (hours) for Qwen2.5-3B under the common ZO training budget.
| Methods | ARC-C Mem. | ARC-C Iter/s | HellaSwag Mem. | HellaSwag Iter/s |
| FT | 600.5 | 0.126 | 664.8 | 0.074 |
| LoRA | 256.0 | 0.161 | 282.2 | 0.105 |
| MeZO | 97.29 | 0.174 | 104.8 | 0.104 |
| WIS-MeZO | 122.0 | 0.169 | 132.3 | 0.102 |
| WIS-MeZO (1-bit) | 98.84 | 0.160 | 107.7 | 0.099 |
Table 6. Peak GPU memory (GB, batch size=16) and iteration speed (iteration/s) for fine-tuning Qwen2.5-32B.
On the 3B model, WIS-MeZO uses less peak memory and has higher throughput than the evaluated Sparse-MeZO implementation. Its wall-clock times on SST-2, ReCoRD, and CB are 1.76, 4.18, and 5.73 hours. These are 2.0%–5.4% lower than Sparse-MeZO and 2.1%–10.7% higher than MeZO. The comparison quantifies the computational cost of masked updates alongside the validation-loss curves indexed by optimization step.
At 32B, a 1-bit mask reduces WIS-MeZO’s memory overhead relative to MeZO to below 3GB on both tasks while keeping similar throughput. This compact representation supports larger-scale sparse ZO fine-tuning.
Study on Density Ratio-
The density ratio sets the number of coordinates available for perturbation. Table 7 compares the evaluated densities on Qwen2.5-3B.
| Ratio- | ARC-E | ARC-C | HellaSwag | WinoGrande | OpenBookQA |
| 0.05 | 85.0 | 57.8 | 71.8 | 68.6 | 54.0 |
| 0.1 | 85.4 | 60.8 | 74.4 | 69.7 | 52.8 |
| 0.2 | 83.8 | 59.5 | 72.0 | 69.5 | 53.6 |
| 0.4 | 84.2 | 59.8 | 71.4 | 67.1 | 52.8 |
| MeZO | 82.9 | 57.8 | 69.9 | 66.3 | 49.0 |
Table 7. The effects of density ratio- for Fine-tuning Qwen2.5-3B with WIS-MeZO.
WIS-MeZO matches or improves on MeZO at every reported task–density combination. A density of gives the highest score on four of the five tasks in this table, while OpenBookQA performs best at .
Effects of Calibration Size and Computation Cost
Table 8 studies calibration-set size on Qwen2.5-3B. Performance improves with larger calibration sets and stabilizes around 128 examples, which we use by default. Table 9 reports the total calibration time. With 128 examples, mask construction takes 0.53–1.26 seconds and is performed once for fixed-mask WIS-MeZO.
| Calibration size | ARC-E | ARC-C | HellaSwag | WinoGrande | OpenBookQA |
| 32 | 85.0 | 60.5 | 74.0 | 67.4 | 54.0 |
| 64 | 85.2 | 60.5 | 74.3 | 67.8 | 53.0 |
| 128 (default) | 85.4 | 60.8 | 74.4 | 69.7 | 54.0 |
| 256 | 85.4 | 60.5 | 74.6 | 69.7 | 54.0 |
| 512 | 85.9 | 60.8 | 74.5 | 69.7 | 53.9 |
Table 8. Effects of the calibration size on WIS-MeZO fine-tuning of Qwen2.5-3B.
| Calibration size | ARC-E | ARC-C | HellaSwag | WinoGrande | OpenBookQA |
| 32 | 0.32 | 0.33 | 0.43 | 0.26 | 0.27 |
| 64 | 0.49 | 0.49 | 0.72 | 0.34 | 0.38 |
| 128 (default) | 0.80 | 0.89 | 1.26 | 0.53 | 0.60 |
| 256 | 1.49 | 1.55 | 2.36 | 0.91 | 1.00 |
| 512 | 2.70 | 2.98 | 4.59 | 1.66 | 1.84 |
Table 9. Total calibration time (seconds) for Qwen2.5-3B.
Conclusion
We present WIS-MeZO, which uses downstream Wanda scores to select sparse perturbation coordinates for memory-efficient LLM fine-tuning. Our gradient-energy analysis explains the importance of support selection, while experiments across 16 benchmarks and models from 3B to 32B show improved aggregate performance over MeZO. By combining forward-only updates with one-time calibration and compact mask storage, WIS-MeZO also offers a promising direction for resource-constrained on-device adaptation.
Proofs
Notation and Gradient Energy
Table 10 summarizes the notation. The support-selection analysis considers one linear layer. The alignment and convergence results use the total selected dimension across the optimized parameter blocks.
| Symbol | Description |
| Weights, binary support mask, and Gaussian perturbation. | |
| Stochastic gradient, finite-difference estimate, and its directional limit. | |
| Per-observation input and backpropagated signal. | |
| Matrix of calibration activation observations for one layer. | |
| Population and empirical second moments of feature . | |
| Wanda score of weight . | |
| WIS support, oracle support, and their union in row . | |
| Perturbation density and selected coordinates per row of a layer. | |
| Total number of parameters in the optimized blocks. | |
| Total number of selected perturbation coordinates. | |
| Bounds on input–error dependence. | |
| Normalized sub-Gaussian constant and calibration error tolerance. | |
| Row-wise comparison factors and their minimum. | |
| Captured energy, oracle capture, and a trajectory-wise lower bound. | |
| Gradient–saliency comparison constants. | |
| Perturbation radius, learning rate, and mask refresh interval. | |
| Smoothness, PL, and stochastic-variance constants. | |
| Gaussian moment factor and variance bound including finite differences. | |
| Euclidean and Frobenius norms. | |
| Frobenius inner product and element-wise multiplication. |
Table 10. Summary of notation.
For a linear layer , the per-example gradient satisfies
The calibration analysis concerns over the downstream distribution at a fixed parameter value. For calibration observations,
The following proofs establish the energy-capture and convergence results in the main text.
Proof of Lemma 5.3
Proof. Assumption 5.1 gives
Under Assumption 5.2, the squared activation is sub-exponential with scale bounded by a constant multiple of . For features with , concentration of the empirical mean gives, for a universal constant and ,
Features with satisfy almost surely. A union bound over the features yields
with probability at least when . On this event,
For every nonzero-energy row and , combining Eqs. equation (2) and equation (3) gives
The two coefficients define and , and
Rows with zero gradient energy contribute zero to the global ratio.
Proof of Theorem 5.4
Proof. Define the row-wise oracle over the full row by
The WIS support maximizes the sum of squared saliency scores among all sets of size . On the concentration event in Lemma 5.3, both supports belong to , so
Dividing by the total expected energy in the row and summing with weights gives
For a uniform row-wise mask independent of the gradient, each coordinate is included with probability . Therefore,
The sufficient condition then gives the stated comparison.
Proof of Lemma 5.5
Proof. Fix the parameter value and mask. Write , where the entries of are independent standard Gaussians, and . Taking the perturbation expectation first gives
The Gaussian fourth-moment identity in the selected coordinates yields
Hence,
Substituting Eqs. equation 4 and equation (5) into the moment-normalized alignment gives
For a uniform random mask independent of this gradient distribution, taking an additional expectation over masks replaces by . The corresponding ratio of moment-normalized alignment values is .
Proof of Theorem 5.8
Proof. Condition on the fixed calibration mask and the optimization history before iteration . Write for expectation over the current mini-batch and perturbation, and define
The actual central-difference estimator and its directional limit are
Gaussian symmetry and integration by parts give
For , smoothness and imply
For each mini-batch, the fundamental theorem of calculus and Assumption 5.6 give
Since has a chi-squared distribution with degrees of freedom,
By Lemma 5.5 and the conditional variance bound,
Combining this inequality with Eq. equation 7 and yields
Both the estimator and its bias are supported on . Applying Young’s inequality gives
Using and population smoothness,
For , substituting Eqs. equation 6, equation 8, and equation 9 gives
Now set . Taking full expectations, summing, and using and the deterministic lower bound on captured energy yield
Finally, for , so
Convergence under the PL Condition
The same descent inequality gives geometric convergence to a noise and finite-difference neighborhood under a PL condition.
Assumption A.1 (-PL Condition). For ,
Theorem A.2 (Convergence under the PL Condition). Suppose Assumptions 5.6, 5.7, and A.1 hold, together with the fixed-support energy condition in Theorem 5.8. Let , retain from that theorem, and take with . Then
Proof. Write . Substituting the constant step size into Eq. equation 10 and applying the support-energy and PL conditions gives
Set . Iterating this inequality yields
Experimental Details
For the GLUE, SQuAD, and SuperGLUE experiments, all ZO methods train for 20,000 steps with a batch size of 16; FT and LoRA use a batch size of 8. Checkpoints are selected by validation performance. We use a sequence length of 2048, constant learning rates, and zero weight decay for ZO methods, following the settings in MeZO and Sparse-MeZO [23, 21].
Table 11–Table 13 report the hyperparameter search and selected configurations. Results on GLUE and SQuAD are averaged over five seeds, while SuperGLUE results are from one run. WIS-MeZO uses 128 calibration examples from downstream training data and a fixed mask.
| Methods | Hyperparameters | Values |
| FT | Batch size | 8 |
| FT | Learning rate | |
| FT | ||
| LoRA | Batch size | 8 |
| LoRA | Learning rate | |
| LoRA | Rank | 8 |
| LoRA | Alpha- | 16 |
| LoRA | ||
| MeZO | Batch size | 16 |
| MeZO | Learning rate | |
| MeZO | ||
| LOZO | Batch size | 16 |
| LOZO | Learning rate | |
| LOZO | ||
| LOZO | Rank | |
| LOZO | Interval | 50 |
| Sparse-MeZO | Batch size | 16 |
| Sparse-MeZO | Learning rate | |
| Sparse-MeZO | ||
| Sparse-MeZO | Density ratio- | |
| Random-MeZO | Batch size | 16 |
| Random-MeZO | Learning rate | |
| Random-MeZO | ||
| Random-MeZO | Density ratio- | |
| WIS-MeZO | Batch size | 16 |
| WIS-MeZO | Learning rate | |
| WIS-MeZO | ||
| WIS-MeZO | Density ratio- | |
| WISD-MeZO | Batch size | 16 |
| WISD-MeZO | Learning rate | |
| WISD-MeZO | ||
| WISD-MeZO | Density ratio- | |
| WISD-MeZO | Frequency- |
Table 11. The hyperparameter grids used for the experiments in the main text.
| Models | Methods | CB | MultiRC | WSC | ReCoRD | BoolQ |
| LLaMA3.2-3B | FT | 1.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-6 | 1.0e-5 |
| LLaMA3.2-3B | LoRA | 5.0e-4 | 1.0e-4 | 5.0e-4 | 1.0e-5 | 5.0e-4 |
| LLaMA3.2-3B | MeZO | 2.5e-7 | 5.0e-7 | 2.5e-7 | 2.5e-7 | 1.0e-7 |
| LLaMA3.2-3B | LOZO | 1.0e-7 | 2.5e-7 | 1.0e-7 | 2.5e-7 | 1.0e-7 |
| LLaMA3.2-3B | Sparse-MeZO | 1.0e-6, 0.4 | 1.0e-6, 0.2 | 5.0e-6, 0.4 | 1.0e-6, 0.2 | 2.5e-6, 0.2 |
| LLaMA3.2-3B | Random-MeZO | 5.0e-7, 0.2 | 5.0e-7, 0.2 | 2.5e-7, 0.1 | 5.0e-7, 0.4 | 5.0e-7, 0.2 |
| LLaMA3.2-3B | WIS-MeZO | 2.5e-7, 0.1 | 2.5e-7, 0.1 | 5.0e-7, 0.05 | 1.0e-7, 0.2 | 1.0e-7, 0.05 |
| LLaMA3.2-3B | WISD-MeZO | 2.5e-7, 0.1, 2000 | 2.5e-7, 0.1, 5000 | 1.0e-7, 0.05, 2000 | 1.0e-7, 0.2, 10000 | 1.0e-7, 0.1, 10000 |
| Qwen2.5-3B | FT | 5.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-6 | 1.0e-5 |
| Qwen2.5-3B | LoRA | 5.0e-5 | 5.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-4 |
| Qwen2.5-3B | MeZO | 5.0e-7 | 2.5e-7 | 2.5e-7 | 5.0e-7 | 2.5e-7 |
| Qwen2.5-3B | LOZO | 1.0e-7 | 1.0e-7 | 1.0e-7 | 2.5e-7 | 1.0e-7 |
| Qwen2.5-3B | Sparse-MeZO | 2.5e-6, 0.2 | 1.0e-6, 0.4 | 2.5e-6, 0.2 | 5.0e-6, 0.2 | 2.5e-6, 0.2 |
| Qwen2.5-3B | Random-MeZO | 1.0e-7, 0.1 | 5.0e-7, 0.4 | 2.5e-7, 0.2 | 5.0e-7, 0.2 | 1.0e-7, 0.1 |
| Qwen2.5-3B | WIS-MeZO | 1.0e-7, 0.1 | 5.0e-8, 0.05 | 1.0e-7, 0.2 | 1.0e-7, 0.1 | 1.0e-7, 0.2 |
| Qwen2.5-3B | WISD-MeZO | 2.5e-7, 0.2, 2000 | 2.5e-7, 0.4, 5000 | 1.0e-7, 0.2, 10000 | 1.0e-7, 0.1, 2000 | 1.0e-7, 0.2,10000 |
Table 13. Selected hyperparameters for the SuperGLUE experiments.
Regarding the computing environment, we use Python 3.10.14 and PyTorch 2.8.0 [26]. All experiments were conducted on NVIDIA H20 GPUs.
Additional Gradient Energy Capture Results
We compare Random, W-small, W-large, and WIS supports at matched density. The supports are selected uniformly at random, by small weight magnitude, by large weight magnitude, and by the Wanda score, respectively. For each task, we evaluate their gradient energy capture on 2,000 gradient snapshots from an auxiliary AdamW training run on LLaMA-3.2-3B. The supports are fixed at initialization, and all strategies are evaluated on the same gradients.
For a support mask and a mini-batch gradient , energy capture is measured by
We report the average over optimization steps. Random masking captures an expected fraction of gradient energy. At , WIS preserves over 40% of the energy and exceeds the other selection rules. W-small also outperforms W-large and random masking.
Figure 3–Figure 5 report the additional results at density ratios of 0.1, 0.2, and 0.4, respectively. WIS captures the most gradient energy in nearly all reported task–density settings. Its advantage is largest at low densities. At on MNLI, WIS and W-small have comparable capture ratios. W-small consistently captures more energy than W-large and generally exceeds random masking. These comparisons describe how the selected coordinates align with the shared AdamW gradient snapshots and show the contribution of downstream activation information to support selection.
Figure 3. Gradient energy capture ratio on LLaMA-3.2-3B at a density ratio of . Random denotes uniform random masking. W-small and W-large select coordinates with the smallest and largest weight magnitudes, respectively, while WIS selects coordinates using Wanda weight–activation scores.
Figure 5. Gradient energy capture ratio on LLaMA-3.2-3B at a density ratio of . Random denotes uniform random masking. W-small and W-large select coordinates with the smallest and largest weight magnitudes, respectively, while WIS uses Wanda weight–activation scores.
Detailed Pseudocode
This section provides the detailed implementations of the parameter perturbation and WIS mask construction routines used in Algorithm 1. For each linear layer, denotes its weight matrix, denotes the random perturbation, and denotes the corresponding binary mask.
Seed replay regenerates the perturbation for the two loss evaluations and the update, retaining the memory advantages of MeZO.
Algorithm 3 applies the Wanda score and selects the top- coordinates in each row. WIS-MeZO constructs the mask once, while WISD-MeZO refreshes it every steps.
| Models | Methods | MNLI | RTE | QQP | MRPC | SST-2 | SQuAD |
| LLaMA3.2-3B | FT | 1.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-5 |
| LLaMA3.2-3B | LoRA | 5.0e-4 | 5.0e-4 | 5.0e-4 | 5.0e-4 | 5.0e-4 | 5.0e-5 |
| LLaMA3.2-3B | MeZO | 5.0e-7 | 2.5e-7 | 5.0e-7 | 2.5e-7 | 5.0e-7 | 2.5e-7 |
| LLaMA3.2-3B | LOZO | 1.0e-7 | 2.5e-7 | 1.0e-7 | 2.5e-7 | 2.5e-7 | 2.5e-7 |
| LLaMA3.2-3B | Sparse-MeZO | 2.5e-6, 0.2 | 1.0e-6, 0.2 | 1.0e-6, 0.4 | 1.0e-6, 0.4 | 1.0e-6, 0.4 | 1.0e-6, 0.2 |
| LLaMA3.2-3B | Random-MeZO | 5.0e-7, 0.2 | 5.0e-7, 0.2 | 5.0e-7, 0.4 | 5.0e-7, 0.2 | 5.0e-7, 0.2 | 5.0e-7, 0.2 |
| LLaMA3.2-3B | WIS-MeZO | 2.5e-7, 0.2 | 2.5e-7, 0.2 | 5.0e-7, 0.4 | 2.5e-7, 0.1 | 1.0e-7, 0.1 | 1.0e-7, 0.1 |
| LLaMA3.2-3B | WISD-MeZO | 2.5e-7, 0.2, 5000 | 2.5e-7, 0.1, 2000 | 2.5e-7, 0.2, 10000 | 2.5e-7, 0.1, 2000 | 1.0e-7, 0.1, 5000 | 1.0e-7, 0.1, 5000 |
| Qwen2.5-3B | FT | 1.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-5 | 1.0e-5 |
| Qwen2.5-3B | LoRA | 1.0e-4 | 1.0e-4 | 5.0e-5 | 1.0e-4 | 1.0e-5 | 5.0e-4 |
| Qwen2.5-3B | MeZO | 1.0e-7 | 1.0e-7 | 2.5e-7 | 5.0e-7 | 5.0e-7 | 5.0e-7 |
| Qwen2.5-3B | LOZO | 5.0e-8 | 1.0e-7 | 1.0e-7 | 1.0e-7 | 2.5e-7 | 2.5e-7 |
| Qwen2.5-3B | Sparse-MeZO | 2.5e-6, 0.1 | 2.5e-6, 0.4 | 5.0e-6, 0.1 | 5.0e-6, 0.2 | 5.0e-6, 0.4 | 5.0e-6, 0.1 |
| Qwen2.5-3B | Random-MeZO | 1.0e-7, 0.1 | 5.0e-7, 0.4 | 1.0e-7, 0.1 | 2.5e-7, 0.2 | 5.0e-7, 0.4 | 5.0e-7, 0.4 |
| Qwen2.5-3B | WIS-MeZO | 1.0e-7, 0.2 | 1.0e-7, 0.1 | 5.0e-8, 0.4 | 5.0e-8, 0.2 | 2.5e-7, 0.4 | 5.0e-8, 0.05 |
| Qwen2.5-3B | WISD-MeZO | 5.0e-8, 0.2, 5000 | 5.0e-8, 0.05, 10000 | 5.0e-8, 0.4, 2000 | 5.0e-8, 0.2, 5000 | 2.5e-7, 0.2, 2000 | 5.0e-8, 0.05, 5000 |
Table 12. Selected hyperparameters for the GLUE and SQuAD experiments.
Figure 4. Gradient energy capture ratio on LLaMA-3.2-3B at a density ratio of . All mask construction strategies use the same row-wise perturbation density, model parameters, gradient snapshots, and evaluation data.
Algorithm 2 Perturb Parameters
Function PerturbParameters():
Input: Weights , density ratio , perturbation scale , mask , and random seed
Reset the random number generator with seed
Sample with i.i.d. standard Gaussian entries
{In-place perturbation}
return
Algorithm 3 Compute Mask
Function ComputeMask():
Input: Weights , calibration activations , and density ratio
, for
{Selected parameters per row}
for each row in do
, for
end for
return
References
- [1]HanQin Cai, Daniel McKenzie, Wotao Yin, and Zhenliang Zhang. Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization, 32(2): 687–714, 2022.DOI
- [2]Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. arXiv preprint arXiv:2310.02025, 2023.arxiv.org/abs/2310.02025
- [3]Xiangyi Chen, Sijia Liu, Kaidi Xu, Xingguo Li, Xue Lin, Mingyi Hong, and David Cox. Zo-adamm: Zeroth-order adaptive momentum method for black-box optimization. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1910.06513
- [4]Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.07698
- [5]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
- [6]Sizhe Dang, yangyangGuo, Yanjun Zhao, Haishan Ye, xiaodong zheng, Guang Dai, and Ivor Tsang. FZOO: Fast zeroth-order optimizer for fine-tuning large language models towards adam-scale speed. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2506.09034
- [7]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318–30332, 2022.
- [8]Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pages 2943–2952. PMLR, 2020.arxiv.org/abs/1911.11134
- [9]Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR, 2023.arxiv.org/abs/2301.00774
- [10]Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080, 2024.arxiv.org/abs/2404.08080
- [11]Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341–2368, 2013.arxiv.org/abs/1309.5549
- [12]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
- [13]Jiaqi Gu, Chenghao Feng, Zheng Zhao, Zhoufeng Ying, Ray T Chen, and David Z Pan. Efficient on-chip learning for optical neural networks through power-aware sparse zeroth-order optimization. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 7583–7591, 2021.arxiv.org/abs/2012.11148
- [14]Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhou Xu. Zeroth-order fine-tuning of LLMs with transferable static sparsity. In The Thirteenth International Conference on Learning Representations, 2025.
- [15]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
- [16]Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pages 2137–2146. PMLR, 2018.arxiv.org/abs/1804.08598
- [17]Taejong Joo, Wenhan Xia, Cheolmin Kim, Ming Zhang, and Eugene Ie. On surprising effectiveness of masking updates in adaptive optimizers. arXiv preprint arXiv:2602.15322, 2026.arxiv.org/abs/2602.15322
- [18]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.arxiv.org/abs/1412.6980
- [19]Wei Lin, Yining Jiang, Qingyu Song, Qiao Xiang, and Hong Xu. Agzo: Activation-guided zeroth-order optimization for llm fine-tuning. arXiv preprint arXiv:2601.17261, 2026.arxiv.org/abs/2601.17261
- [20]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.09353
- [21]Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse meZO: Less parameters for better performance in zeroth-order LLM fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2402.15751
- [22]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
- [23]Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:50338–50375, 2023.arxiv.org/abs/2305.17333
- [24]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391, 2018.
- [25]Mayumi Ohta, Nathaniel Berger, Artem Sokolov, and Stefan Riezler. Sparse perturbations for improved convergence in stochastic zeroth-order optimization. In International Conference on Machine Learning, Optimization, and Data Science, pages 39–64. Springer, 2020.arxiv.org/abs/2006.01759
- [26]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1912.01703
- [27]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
- [28]James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332–341, 2002.
- [29]Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2306.11695
- [30]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pages 353–355, 2018.
- [31]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1905.00537
- [32]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Daiyiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Minfeng Xue, Pei Zhang, Qin Zhu, Rui Meng, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.arxiv.org/abs/2412.15115
- [33]Ziming Yu, Pan Zhou, Sike Wang, Jia Li, Mi Tian, and Hua Huang. Zeroth-order fine-tuning of llms in random subspaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4475–4485, 2025.arxiv.org/abs/2410.08989
- [34]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019.arxiv.org/abs/1905.07830
- [35]Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: a benchmark. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2402.11592
- [36]Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, and Zheng Zhang. Mazo: Masked zeroth-order optimization for multi-task fine-tuning of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18537–18554, 2025.
- [37]Hanzhen Zhao, Shihong Ding, Cong Fang, and Zhouchen Lin. Pazo: Preconditioned accelerated zeroth-order optimization for fine-tuning llms. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.DOI
- [38]Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024.arxiv.org/abs/2403.03507
- [39]Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. APOLLO: SGD-like memory, adamw-level performance. In Eighth Conference on Machine Learning and Systems, 2025.arxiv.org/abs/2412.05270




