betaFeedback

WorksComputer Science

Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization

Abstract

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical. We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight–activation saliency to construct task-conditioned sparse perturbation masks while keeping the pretrained model dense. We analyze support quality through gradient energy capture and relate it to gradient estimation and convergence. Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps. Comparisons with random and magnitude-based sparse perturbations further highlight the importance of informative support selection.

Introduction

Fine-tuning large language models (LLMs) requires substantial memory for the activations, gradients, and optimizer states used in backpropagation. Parameter-efficient fine-tuning [15, 20], quantization [7], and memory-efficient optimizers [38, 39] reduce different parts of this cost. Zeroth-order (ZO) optimization offers another approach by estimating update directions through forward evaluations, as demonstrated by MeZO [23]. Its reduced training-memory requirements are particularly appealing for on-device adaptation, where models must be personalized under limited device resources [14].

Despite this memory advantage, ZO optimization remains challenging in the high-dimensional parameter spaces of LLMs. The variance of random-direction gradient estimates grows with the number of perturbed coordinates, making it difficult to obtain informative updates. Sparse ZO methods address this issue by restricting perturbations to a subset of parameters. However, reducing the number of coordinates is only part of the problem: the selected support must also retain useful information for downstream optimization. Existing methods select small-magnitude weights [21] or transfer sensitivity information from pre-training [14]. This motivates selecting sparse supports using information available from the downstream task itself.

We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight–activation saliency [29] to select task-conditioned perturbation coordinates. A lightweight forward calibration produces a row-wise sparse mask that is reused during MeZO updates, while the pretrained model remains dense. In this way, downstream activation information guides the allocation of a fixed perturbation budget.

To understand the effect of coordinate selection, we study gradient energy capture, which measures the proportion of gradient energy contained in the selected support. This perspective connects support quality to the alignment of ZO estimates and to convergence. We complement the analysis with comparisons against alternative sparse supports and evaluations across downstream tasks and model scales.

Our main contributions are as follows.

  • Task-Conditioned Sparse Perturbations. We introduce WIS-MeZO, which selects sparse perturbation coordinates with downstream Wanda saliency while retaining the dense pretrained model and requiring only forward calibration.

  • Gradient-Energy Analysis. We analyze the relationship between saliency-based support selection and captured gradient energy, and establish theoretical results connecting support quality to ZO estimation and convergence.

  • Empirical Evaluation. Experiments across 16 benchmarks and LLaMA and Qwen models up to 32B parameters show improved aggregate performance over MeZO. Comparisons with random and magnitude-based sparse perturbations, together with validation-loss curves, further demonstrate the benefit of informed support selection.

Related Work

Zeroth-order optimization estimates update directions from function evaluations instead of backpropagation [28, 16]. Classical approaches include ZO-SGD [11] and ZO-Adam [3], while MeZO [23] enables memory-efficient LLM fine-tuning through seed-replayed perturbations. Subsequent work improves ZO estimation through variance reduction [10], preconditioning [37], additional queries [6], and low-rank or structured perturbations [36, 33, 4]. AGZO [19] also uses downstream activations, constructing low-rank perturbation subspaces from current mini-batches. A complementary approach is to reduce the search dimension by selecting a sparse set of weight coordinates.

Sparse coordinate selection has been studied in ZO optimization [25, 13, 1, 35]. DeepZero [2] exploits finite-difference information, Sparse-MeZO [21] selects small-magnitude weights, and SensZOQ [14] uses sensitivity information from pre-training. Separately, Wanda [29] and SparseGPT [9] study weight importance for pruning, while dynamic sparse training [8, 17] adapts sparse networks during optimization. Building on Wanda’s weight–activation score, WIS-MeZO uses downstream data to select perturbation coordinates while keeping the model dense. We analyze the quality of this support through gradient energy capture.

Preliminaries

We first review the MeZO estimator and the Wanda score, which together form the basis of WIS-MeZO.

Zeroth-Order Optimization

For a linear layer, let W∈Rm×n\mathbf{W} \in\mathbb{R}^{m \times n} denote the weight matrix and let G=∇f(W,B)\mathbf{G} = \nabla f(\mathbf{W}, \mathcal{B}) be its gradient on a training mini-batch B\mathcal{B}. MeZO [23] estimates this gradient using a random direction Z\mathbf{Z} and a perturbation radius ϵ>0\epsilon> 0,

G^=f(W+ϵZ,B)−f(W−ϵZ,B)2ϵZ.\widehat{\mathbf{G}} = \frac{f(\mathbf{W}+\epsilon\mathbf{Z}, \mathcal{B}) - f(\mathbf{W}-\epsilon\mathbf{Z}, \mathcal{B})}{2\epsilon}\mathbf{Z}.

The entries of Z\mathbf{Z} are independent standard Gaussian variables. Each estimate uses one direction and two loss evaluations on the same mini-batch. As ϵ→0\epsilon\to0, its expectation over Z\mathbf{Z} approaches G\mathbf{G}. The update is Wt+1=Wt−ηtG^t\mathbf{W}^{t+1} = \mathbf{W}^{t} - \eta_t \widehat{\mathbf{G}}^{t}.

The matrix notation describes a single layer with mnmn coordinates. For full-model updates, W\mathbf{W} and the corresponding masks denote the collections of parameter blocks being optimized. Their inner products and squared Frobenius norms sum over these blocks. We use

d=∑ℓmℓnℓ,deff=∑ℓmℓkℓd = \sum_{\ell} m_{\ell}n_{\ell}, \qquad d_{\mathrm{eff}} = \sum_{\ell} m_{\ell}k_{\ell}

for the total candidate dimension and the number of selected coordinates, respectively. A common row-wise density r=kℓ/nℓr = k_\ell/n_\ell gives deff=rdd_{\mathrm{eff}} = rd.

Wanda Importance Score

Wanda [29] scores each weight by its magnitude and the activation norm of its input feature. For a weight matrix W\mathbf{W} and calibration activations X\mathbf{X}, the score is

Si,j=∣Wi,j∣∥Xj,:∥2.S_{i,j} = \lvert W_{i,j} \rvert\lVert X_{j,:} \rVert_2.

WIS-MeZO uses this score to select the perturbation mask.

Methodology

Building on these two ingredients, WIS-MeZO uses Wanda scores to choose perturbation coordinates before applying MeZO updates on the selected support. We describe the support construction and optimization procedure below.

WIS-Guided Sparse Perturbation

For each layer W∈Rm×n\mathbf{W} \in\mathbb{R}^{m \times n}, let M∈{0,1}m×n\mathbf{M} \in\{0,1\}^{m \times n} be a binary mask. Given a density r=k/nr = k/n, WIS-MeZO selects the kk largest scores in each row of S\mathbf{S}. The mask records which coordinates receive perturbations and updates; the underlying weight matrix remains dense. Row-wise selection allocates the same perturbation budget to every output unit.

The perturbation direction is U=(Z⊙M)/r\mathbf{U} = (\mathbf{Z} \odot\mathbf{M})/\sqrt{r}, with independent standard Gaussian entries in Z\mathbf{Z}. For one layer, this gives E∥U∥F2=mn\mathbb{E}\lVert\mathbf{U}\rVert_F^2 = mn while restricting the support to mkmk coordinates. Across the model, the corresponding support dimension is deffd_{\mathrm{eff}}.

Calibration from Downstream Data

WIS-MeZO constructs the mask from 128 examples of the downstream training data using Wanda scores. This calibration is performed once before optimization; WISD-MeZO periodically refreshes the mask every τ\tau steps. Appendix § 7.3 examines the effect and cost of calibration size.

The WIS-MeZO Algorithm

In each iteration tt, we sample a single random seed to generate the perturbation Zt\mathbf{Z}^{t}. The gradient estimator uses one sampled direction and two loss evaluations on the same training mini-batch,

G^t=f(Wt+εZt⊙Mr,B)−f(Wt−εZt⊙Mr,B)2ε⋅Zt⊙Mr.\widehat{\mathbf{G}}^{t} = \frac{f\left(\mathbf{W}^{t} + \varepsilon\frac{\mathbf{Z}^{t} \odot\mathbf{M}}{\sqrt{r}}, \mathcal{B}\right) - f\left(\mathbf{W}^{t} - \varepsilon\frac{\mathbf{Z}^{t} \odot\mathbf{M}}{\sqrt{r}}, \mathcal{B}\right)}{2\varepsilon} \cdot\frac{\mathbf{Z}^{t} \odot\mathbf{M}}{\sqrt{r}}.

Algorithm 1 WIS-MeZO

  1. Input: Initial weights W0\mathbf{W}^{0}, step sizes {ηt}\{\eta_t\}, perturbation scale ϵ\epsilon, mask update frequency τ\tau, density ratio rr, calibration dataset Dcal\mathcal{D}_{\mathrm{cal}}

  2. for step t=0,1,2,…,T−1t = 0,1,2,\ldots,T-1 do

  3. if t mod τ=0t \bmod\tau= 0 then

  4. Fetch activations X∼Dcal\mathbf{X} \sim\mathcal{D}_{\mathrm{cal}} {Collect calibration activations}

  5. M←ComputeMask⁡(Wt,X,r)\mathbf{M} \leftarrow\operatorname{ComputeMask}(\mathbf{W}^{t},\mathbf{X},r) {Update mask using Wanda scores}

  6. end if

  7. Sample training batch B\mathcal{B} and random seed srngs_{\mathrm{rng}}

  8. Wt←PerturbParameters⁡(Wt,r,ϵ,M,srng)\mathbf{W}^{t} \leftarrow\operatorname{PerturbParameters}(\mathbf{W}^{t},r,\epsilon,\mathbf{M},s_{\mathrm{rng}})

  9. L+←f(Wt,B)L_{+} \leftarrow f(\mathbf{W}^{t},\mathcal{B})

  10. Wt←PerturbParameters⁡(Wt,r,−2ϵ,M,srng)\mathbf{W}^{t} \leftarrow\operatorname{PerturbParameters}(\mathbf{W}^{t},r,-2\epsilon,\mathbf{M},s_{\mathrm{rng}}) {Move to the negative perturbation}

  11. L−←f(Wt,B)L_{-} \leftarrow f(\mathbf{W}^{t},\mathcal{B})

  12. Wt←PerturbParameters⁡(Wt,r,ϵ,M,srng)\mathbf{W}^{t} \leftarrow\operatorname{PerturbParameters}(\mathbf{W}^{t},r,\epsilon,\mathbf{M},s_{\mathrm{rng}}) {Recover parameters}

  13. Reset the random number generator with seed srngs_{\mathrm{rng}} and regenerate Z\mathbf{Z}

  14. G^t←L+−L−2ϵrZ⊙M\widehat{\mathbf{G}}^{t} \leftarrow\frac{L_{+}-L_{-}}{2\epsilon\sqrt{r}}\mathbf{Z}\odot\mathbf{M} {ZO gradient estimate}

  15. Wt+1←Wt−ηtG^t\mathbf{W}^{t+1} \leftarrow\mathbf{W}^{t}-\eta_t\widehat{\mathbf{G}}^{t} {Update the parameters}

  16. end for

Here, ⊙\odot denotes element-wise multiplication, and the update is Wt+1=Wt−ηtG^t\mathbf{W}^{t+1}=\mathbf{W}^{t}-\eta_t\widehat{\mathbf{G}}^{t}. Seed replay regenerates perturbations for in-place updates without storing full-model gradients. Algorithm 1 summarizes WIS-MeZO, with supplementary pseudocode in Appendix § 12.

With in-place perturbations and seed replay, the main additional storage is the binary mask. A bit-packed representation reduces its cost from 1 byte to 1 bit per candidate parameter, requiring less than 4GB for the 32B model. Together with one-time calibration, this compact storage makes WIS-MeZO a promising approach for memory-constrained on-device adaptation. § 7.1 reports peak GPU memory and throughput.

Theoretical Analysis

Sparse perturbations reduce the search dimension, but different supports retain different amounts of gradient information. We analyze this trade-off through gradient energy capture, first relating Wanda scores to gradient energy and then studying estimator alignment and convergence.

Energy Analysis

For the analysis, let Gi,j=δixjG_{i,j}=\delta_i x_j be the per-example gradient, with δi\delta_i denoting the backpropagated signal and xjx_j an input activation. Write vj=E[xj2]v_j=\mathbb{E}[x_j^2] and v^j\hat{v}_j for the population and empirical activation second moments.

Assumption 5.1 (Bounded Input–Error Dependence). There exist constants 0<γ≤ν<∞0<\gamma\le\nu<\infty such that

γE[δi2]vj≤E[Gi,j2]≤νE[δi2]vj.\gamma\mathbb{E}[\delta_i^2]v_j \le\mathbb{E}[G_{i,j}^2] \le\nu\mathbb{E}[\delta_i^2]v_j.

The constants γ\gamma and ν\nu describe the dependence between squared inputs and squared error signals. Independence gives γ=ν=1\gamma=\nu=1.

Assumption 5.2 (Calibration Concentration). At the parameter value used for calibration, the BB activation observations are independent draws from the downstream distribution. For every input feature,

∥xj∥ψ22≤Kvj,K≥1,\lVert x_j\rVert_{\psi_2}^2 \le K v_j,\qquad K\ge1,

where ∥⋅∥ψ2\lVert\cdot\rVert_{\psi_2} is the sub-Gaussian norm.

Under this assumption, B=Ω(K2log⁡n/ξ2)B=\Omega(K^2\log n/\xi^2) gives simultaneous relative accuracy ∣v^j−vj∣≤ξvj|\hat{v}_j-v_j|\le\xi v_j over the nn input features with probability at least 1−O(n−1)1-O(n^{-1}), for ξ∈(0,1)\xi\in(0,1).

Let IWIS,i\mathcal{I}_{\mathrm{WIS},i} be the WIS-selected support in row ii, and let Ii∗\mathcal{I}_i^* contain the kk largest expected gradient energies among all nn coordinates in that row. Set Ai=IWIS,i∪Ii∗\mathcal{A}_i=\mathcal{I}_{\mathrm{WIS},i}\cup\mathcal{I}_i^*. For rows with nonzero gradient energy, suppose

wmin⁡,i=min⁡j∈AiWi,j2>0,wmax⁡,i=max⁡j∈AiWi,j2.w_{\min,i}=\min_{j\in\mathcal{A}_i}\mathbf{W}_{i,j}^{2}>0,\qquad w_{\max,i}=\max_{j\in\mathcal{A}_i}\mathbf{W}_{i,j}^{2}.

Define

κi=γwmin⁡,iνwmax⁡,i,κi,ξ=κi1−ξ1+ξ,κξ=min⁡i:ωi>0κi,ξ.\kappa_i=\frac{\gamma w_{\min,i}}{\nu w_{\max,i}},\qquad\kappa_{i,\xi}=\kappa_i\frac{1-\xi}{1+\xi},\qquad\kappa_\xi=\min_{i:\omega_i>0}\kappa_{i,\xi}.

These factors account for input–error dependence, weight variation within the compared supports, and calibration error.

Lemma 5.3 (Gradient-Saliency Proxy Bound). Under Assumptions 5.1 and 5.2, if B=Ω(K2log⁡n/ξ2)B=\Omega(K^2\log n/\xi^2), then with probability at least 1−O(n−1)1-O(n^{-1}),

Ci,lowSi,j2≤E[Gi,j2]≤Ci,highSi,j2C_{i,\mathrm{low}}\mathbf{S}_{i,j}^{2}\leq\mathbb{E}[\mathbf{G}_{i,j}^{2}]\leq C_{i,\mathrm{high}}\mathbf{S}_{i,j}^{2}

simultaneously for j∈Aij\in\mathcal{A}_i, where Ci,low/Ci,high=κi,ξC_{i,\mathrm{low}}/C_{i,\mathrm{high}}=\kappa_{i,\xi}.

The lemma relates the Wanda ranking to expected gradient energy.

For a fixed support and nonzero expected gradient energy, define its energy capture ratio by

ρ≜E[∥M⊙G∥F2]E[∥G∥F2]=∑i=1mωiρi,(1)\rho\triangleq\frac{\mathbb{E}[\lVert\mathbf{M}\odot\mathbf{G}\rVert_{F}^{2}]}{\mathbb{E}[\lVert\mathbf{G}\rVert_{F}^{2}]}=\sum_{i=1}^{m}\omega_i\rho_i, \tag*{(1)}

where

ωi=E[∥Gi,:∥22]E[∥G∥F2],ρi=E[∥Mi,:⊙Gi,:∥22]E[∥Gi,:∥22].\omega_i=\frac{\mathbb{E}[\lVert\mathbf{G}_{i,:}\rVert_{2}^{2}]}{\mathbb{E}[\lVert\mathbf{G}\rVert_{F}^{2}]},\qquad\rho_i=\frac{\mathbb{E}[\lVert\mathbf{M}_{i,:}\odot\mathbf{G}_{i,:}\rVert_{2}^{2}]}{\mathbb{E}[\lVert\mathbf{G}_{i,:}\rVert_{2}^{2}]}.

Rows with zero gradient energy contribute zero to the sum. Let ρ∗=∑iωiρi∗\rho^*=\sum_i\omega_i\rho_i^* be the capture ratio of the row-wise oracle supports Ii∗\mathcal{I}_i^*.

Theorem 5.4 (Energy Capture Relative to the Oracle). Under the conditions of Lemma 5.3, with probability at least 1−O(n−1)1-O(n^{-1}),

ρWIS≥∑iωiκi,ξρi∗≥κξρ∗.\rho_{\mathrm{WIS}}\geq\sum_i\omega_i\kappa_{i,\xi}\rho_i^*\geq\kappa_\xi\rho^*.

A uniform row-wise random mask of density rr has expected capture ratio rr at the same parameter value. Thus, κξρ∗>r\kappa_\xi\rho^*>r is a sufficient condition for WIS to exceed this random-mask reference.

The result separates gradient-energy concentration, measured by ρ∗\rho^*, from the accuracy of the saliency ranking, measured by κξ\kappa_\xi. Calibration affects the latter through ξ\xi.

Figure 2 compares support rules on common mini-batch gradient snapshots from an auxiliary AdamW run. All masks are constructed at step 0 and remain fixed. At r=0.05r = 0.05, the plotted WIS supports retain over 40% of the gradient energy, compared with the 5% random-mask expectation. WIS also captures more energy than W-small and W-large in these diagnostics. W-small exceeds random masking, while W-large falls below it, so the observed energy ordering is not determined by increasing weight magnitude. These measurements evaluate how the initial supports align with gradients encountered during the diagnostic run. Results at additional densities appear in Appendix § 11.

Figure 2. Gradient energy capture ratio ρ\rho on LLaMA-3.2-3B at a density ratio of r=0.05r=0.05. We compare uniform random masking, the small-magnitude mask used to represent the selection principle of Sparse-MeZO (W-small), the corresponding large-magnitude mask (W-large), and the Wanda-based WIS mask.

We next consider estimator alignment at a fixed parameter value. For the mini-batch gradient G\mathbf{G}, define ρ\rho by Eq. equation 1 using the mini-batch distribution. Let s=deffs = d_{\mathrm{eff}} be the total number of selected coordinates and U=(Z⊙M)/r\mathbf{U} = (\mathbf{Z} \odot\mathbf{M})/\sqrt{r}, with standard Gaussian entries in Z\mathbf{Z}.

Lemma 5.5 (Moment-Normalized Alignment). For the directional-derivative estimator G^0=⟨G,U⟩U\widehat{\mathbf{G}}_0 = \langle\mathbf{G}, \mathbf{U}\rangle\mathbf{U} with ρ>0\rho> 0,

Align⁡(G^0,G)≔E[⟨G^0,G⟩]E[∥G^0∥F2]E[∥G∥F2]=ρs+2.\operatorname{Align}(\widehat{\mathbf{G}}_0,\mathbf{G}) \coloneqq\frac{\mathbb{E}[\langle\widehat{\mathbf{G}}_0,\mathbf{G}\rangle]}{\sqrt{\mathbb{E}[\lVert\widehat{\mathbf{G}}_0\rVert_F^2]\mathbb{E}[\lVert\mathbf{G}\rVert_F^2]}} = \sqrt{\frac{\rho}{s+2}}.

At the same gradient distribution, averaging over uniform row-wise masks replaces ρ\rho by rr.

The estimator G^0\widehat{\mathbf{G}}_0 is the ϵ→0\epsilon\to0 limit of the central-difference estimator. The lemma isolates the contribution of support quality at a fixed dimension. The next result includes the finite perturbation radius used by the algorithm.

Convergence Analysis

We analyze the fixed support produced at calibration. In this subsection, W\mathbf{W}, M\mathbf{M}, and G\mathbf{G} denote the full collections of parameter blocks, and squared norms sum over all blocks.

Assumption 5.6 (LL-Smooth Mini-Batch Losses). Each mini-batch loss f(⋅,B)f(\cdot,\mathcal{B}) is differentiable and LL-smooth. The population objective f=EB[f(⋅,B)]f = \mathbb{E}_{\mathcal{B}}[f(\cdot,\mathcal{B})] is bounded below by f∗f^*.

Assumption 5.7 (Unbiased Gradients with Bounded Variance). Conditioned on the optimization history, Gt=∇f(Wt,Bt)\mathbf{G}^t = \nabla f(\mathbf{W}^t,\mathcal{B}^t) satisfies

Et[Gt]=∇f(Wt),Et[∥Gt−∇f(Wt)∥F2]≤σ2.\mathbb{E}_t[\mathbf{G}^t] = \nabla f(\mathbf{W}^t), \qquad\mathbb{E}_t[\lVert\mathbf{G}^t-\nabla f(\mathbf{W}^t)\rVert_F^2] \le\sigma^2.

The perturbation is independent of the current mini-batch.

Theorem 5.8 (Finite-Difference Convergence). Suppose Assumptions 5.6 and 5.7 hold. For a fixed mask, let s=deffs = d_{\mathrm{eff}}, a=s+2a = s + 2, Δf=f(W0)−f∗\Delta f = f(\mathbf{W}^0)-f^*, and

σϵ2≜σ2+L2ϵ2s(s+4)4r.\sigma_\epsilon^2 \triangleq\sigma^2 + \frac{L^2\epsilon^2s(s+4)}{4r}.

Assume a deterministic constant ρmin⁡∈(0,1]\rho_{\min} \in(0,1] satisfies

∥M⊙∇f(Wt)∥F2≥ρmin⁡∥∇f(Wt)∥F2\lVert\mathbf{M} \odot\nabla f(\mathbf{W}^t)\rVert_F^2 \ge\rho_{\min}\lVert\nabla f(\mathbf{W}^t)\rVert_F^2

almost surely for 0≤t<T0 \le t < T. With Gaussian perturbations, radius ϵ>0\epsilon> 0, and ηt=r/(4Lat+1)\eta_t = r/(4La\sqrt{t+1}), the central-difference iterates satisfy

1T∑t=0T−1E[∥∇f(Wt)∥F2]≤8LaΔf+12σϵ2(1+log⁡T)ρmin⁡T+4L2ϵ2srρmin⁡.\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\lVert\nabla f(\mathbf{W}^t)\rVert_F^2] \le\frac{8La\Delta f + \frac{1}{2}\sigma_\epsilon^2(1+\log T)}{\rho_{\min}\sqrt{T}} + \frac{4L^2\epsilon^2s}{r\rho_{\min}}.

The convergence bound depends on the selected dimension, the gradient energy retained during training, and the finite-difference radius. The result also highlights the role of support quality in sparse ZO optimization.

Experiments

We now examine how task-conditioned support selection affects downstream fine-tuning. We evaluate five GLUE tasks [30], SQuAD, and five SuperGLUE tasks [31], followed by commonsense and question-answering benchmarks across larger model scales. Additional efficiency and sensitivity results appear in the appendix.

Fine-Tuning on GLUE, SQuAD, and SuperGLUE

We evaluate WIS-MeZO and its dynamic variant WISD-MeZO on LLaMA-3.2-3B [12] and Qwen2.5-3B [32]. All ZO methods use a batch size of 16 and train for 20,000 steps. Hyperparameters are reported in Appendix § 10.

The first-order baselines are full-parameter fine-tuning (FT) with AdamW [18, 22] and LoRA [15]. The ZO baselines are MeZO [23], Sparse-MeZO [21], which selects small-magnitude weights, and Random-MeZO, which uses random sparse supports. These comparisons evaluate the effect of coordinate selection within sparse ZO fine-tuning.

Table 1 and Table 2 report the task results. Figure 1 compares validation loss against optimization steps over five random seeds.

Figure 1. Validation loss versus training steps for fine-tuning LLaMA3.2-3B, averaged over five random seeds. WIS-MeZO reaches comparable validation-loss levels in fewer optimization steps.

Table 1: Results on five GLUE tasks and SQuAD (five random seeds). We report F1 scores for QQP, MRPC, and SQuAD, and accuracy (%) for all other tasks (↑). FT and LoRA denote full parameter fine-tuning and Low-Rank Adaptation, respectively, both utilizing the AdamW optimizer.

ModelsMethodsMNLIRTEQQPMRPCSST-2SQuADAvg.
LLaMA3.2-3BZero-shot38.349.453.778.358.855.055.6
FT82.6±0.682.6 \pm0.687.3±0.887.3 \pm0.879.2±1.079.2 \pm1.086.9±1.086.9 \pm1.095.1±0.595.1 \pm0.588.2±0.488.2 \pm0.486.5
LoRA83.4±0.783.4 \pm0.788.8±0.788.8 \pm0.779.2±1.079.2 \pm1.085.8±1.385.8 \pm1.394.8±0.194.8 \pm0.188.0±0.088.0 \pm0.086.6
MeZO49.4±3.049.4 \pm3.070.3±2.170.3 \pm2.165.0±2.165.0 \pm2.181.4±0.681.4 \pm0.693.9±0.893.9 \pm0.886.7±0.686.7 \pm0.674.4
Sparse-MeZO56.7±2.656.7 \pm2.677.6±3.877.6 \pm3.870.6±0.670.6 \pm0.681.8±0.981.8 \pm0.993.6±0.393.6 \pm0.386.4±0.786.4 \pm0.777.7
Random-MeZO52.1±1.852.1 \pm1.874.3±0.774.3 \pm0.767.4±1.367.4 \pm1.382.6±1.182.6 \pm1.193.9±0.493.9 \pm0.488.1±0.688.1 \pm0.676.4
WIS-MeZO61.2±1.161.2 \pm1.178.3±1.378.3 \pm1.368.7±1.068.7 \pm1.083.1±1.183.1 \pm1.194.0±0.294.0 \pm0.289.2±0.889.2 \pm0.879.1
WISD-MeZO61.4±1.361.4 \pm1.380.1±2.280.1 \pm2.271.2±0.971.2 \pm0.982.9±1.082.9 \pm1.094.0±0.594.0 \pm0.589.4±1.689.4 \pm1.679.8
Qwen2.5-3BZero-shot74.880.534.151.156.083.663.4
FT86.9±0.586.9 \pm0.587.8±0.687.8 \pm0.681.2±0.181.2 \pm0.189.1±0.889.1 \pm0.894.1±0.494.1 \pm0.485.6±0.285.6 \pm0.287.5
LoRA86.6±0.486.6 \pm0.487.9±0.487.9 \pm0.479.5±0.279.5 \pm0.289.6±1.089.6 \pm1.093.6±0.493.6 \pm0.487.2±1.087.2 \pm1.087.4
MeZO84.8±0.784.8 \pm0.784.8±0.984.8 \pm0.978.7±1.178.7 \pm1.184.9±1.884.9 \pm1.892.6±0.292.6 \pm0.286.6±0.986.6 \pm0.985.4
Sparse-MeZO86.1±1.086.1 \pm1.086.6±1.586.6 \pm1.581.3±0.881.3 \pm0.885.9±1.085.9 \pm1.093.6±0.693.6 \pm0.686.3±0.786.3 \pm0.786.6
Random-MeZO85.1±1.385.1 \pm1.385.2±1.685.2 \pm1.681.3±0.681.3 \pm0.684.9±1.184.9 \pm1.193.1±0.293.1 \pm0.286.6±0.686.6 \pm0.686.0
WIS-MeZO86.7±0.486.7 \pm0.487.0±0.787.0 \pm0.781.5±0.681.5 \pm0.685.6±1.185.6 \pm1.193.6±0.293.6 \pm0.287.1±0.887.1 \pm0.886.9
WISD-MeZO86.3±0.986.3 \pm0.987.0±1.087.0 \pm1.081.6±0.781.6 \pm0.786.4±1.286.4 \pm1.293.7±0.193.7 \pm0.186.8±1.186.8 \pm1.187.0

Table 1. Results on five GLUE tasks and SQuAD (five random seeds). We report F1 scores for QQP, MRPC, and SQuAD, and accuracy (%) for all other tasks (↑). FT and LoRA denote full parameter fine-tuning and Low-Rank Adaptation, respectively, both utilizing the AdamW optimizer.

ModelsMethodsCBMultiRCWSCReCoRDBoolQAvg.
LLaMA3.2-3BZero-shot44.647.036.579.761.653.9
FT91.083.657.681.683.179.4
LoRA92.885.262.581.484.381.2
MeZO71.472.461.581.277.672.8
Sparse-MeZO67.879.762.581.481.374.5
Random-MeZO75.074.261.581.078.974.1
WIS-MeZO82.176.363.482.082.677.3
WISD-MeZO82.176.364.481.983.077.6
Qwen2.5-3BZero-shot30.365.863.472.170.360.3
FT96.488.068.474.480.981.6
LoRA94.686.967.476.085.182.0
MeZO96.483.266.373.780.980.1
Sparse-MeZO92.886.167.375.680.180.3
Random-MeZO96.485.665.374.581.680.9
WIS-MeZO96.486.468.274.982.681.7
WISD-MeZO98.286.971.175.682.882.9

Table 2. Experimental results on SuperGLUE. WIS-based methods achieve the strongest aggregate ZO performance on both model families.

WIS-MeZO improves the average score over MeZO, Random-MeZO, and Sparse-MeZO in all four model–benchmark combinations. On LLaMA-3.2-3B, its average scores are 79.1 on the GLUE/SQuAD tasks and 77.3 on SuperGLUE, compared with 77.7 and 74.5 for Sparse-MeZO. The corresponding Qwen2.5-3B scores are 86.9 and 81.7, compared with 86.6 and 80.3. The size of the gain varies across backbones and tasks, with the larger improvements occurring on LLaMA-3.2-3B.

Beyond final task scores, the validation-loss curves in Figure 1 show that WIS-MeZO reaches comparable loss levels in fewer optimization steps than MeZO on RTE, MNLI, and BoolQ. This improvement is consistent with the gradient-energy results in § 5. Runtime and memory measurements are reported in § 7.1.

Periodic recalibration in WISD-MeZO increases the average score in all four model–benchmark combinations. Fixed-mask WIS-MeZO is the primary method, and WISD-MeZO characterizes the effect of refreshing the support during training.

Scaling to Large-Scale LLMs

Having compared support-selection strategies on 3B models, we next examine whether the improvement over MeZO extends to larger models. We evaluate fixed-mask WIS-MeZO on Qwen2.5 models from 3B to 32B parameters using ARC-E, ARC-C [5], HellaSwag [34], WinoGrande [27], and OpenBookQA [24]. Table 3 shows higher average scores at every scale and performance matching or exceeding MeZO on every task. On WinoGrande, WIS-MeZO also improves over the zero-shot model at all four scales.

ModelsMethodsARC-EARC-CHellaSwagWinoGrandeOpenBookQAAvg.
Qwen2.5-3BZero-shot70.048.468.066.340.858.7
MeZO82.957.869.966.349.065.1
WIS-MeZO85.460.874.469.754.068.8
Qwen2.5-7BZero-shot74.952.573.868.943.862.7
MeZO83.059.274.966.551.266.9
WIS-MeZO85.061.279.470.753.870.0
Qwen2.5-14BZero-shot75.453.875.870.643.263.7
MeZO84.067.277.273.857.271.8
WIS-MeZO87.870.981.576.658.875.1
Qwen2.5-32BZero-shot74.554.579.074.343.665.1
MeZO85.969.878.774.952.472.3
WIS-MeZO88.769.884.577.956.275.4

Table 3. Experimental results for Qwen2.5-3B to 32B. WIS-MeZO consistently improves average performance over MeZO across all model scales.

Additional Experimental Results

Memory and Speed Comparison

We report peak GPU memory, iteration throughput, and total training time in Table 4 and Table 5. Table 6 compares standard and bit-packed masks on Qwen2.5-32B.

MethodsSST-2 Mem.SST-2 Iter/sReCoRD Mem.ReCoRD Iter/sCB Mem.CB Iter/s
FT35.53.3787.40.9294.50.75
LoRA22.14.7373.51.3385.00.94
MeZO9.703.5021.51.3945.10.99
Sparse-MeZO16.22.9827.31.3051.00.95
WIS-MeZO11.63.1625.11.3347.00.97

Table 4. Peak GPU memory (GB, batch size=16) and iteration speed (iteration/s) for fine-tuning Qwen2.5-3B.

MethodsSST-2ReCoRDCB
MeZO1.594.005.61
Sparse-MeZO1.864.275.85
WIS-MeZO1.764.185.73

Table 5. End-to-end wall-clock time (hours) for Qwen2.5-3B under the common ZO training budget.

MethodsARC-C Mem.ARC-C Iter/sHellaSwag Mem.HellaSwag Iter/s
FT600.50.126664.80.074
LoRA256.00.161282.20.105
MeZO97.290.174104.80.104
WIS-MeZO122.00.169132.30.102
WIS-MeZO (1-bit)98.840.160107.70.099

Table 6. Peak GPU memory (GB, batch size=16) and iteration speed (iteration/s) for fine-tuning Qwen2.5-32B.

On the 3B model, WIS-MeZO uses less peak memory and has higher throughput than the evaluated Sparse-MeZO implementation. Its wall-clock times on SST-2, ReCoRD, and CB are 1.76, 4.18, and 5.73 hours. These are 2.0%–5.4% lower than Sparse-MeZO and 2.1%–10.7% higher than MeZO. The comparison quantifies the computational cost of masked updates alongside the validation-loss curves indexed by optimization step.

At 32B, a 1-bit mask reduces WIS-MeZO’s memory overhead relative to MeZO to below 3GB on both tasks while keeping similar throughput. This compact representation supports larger-scale sparse ZO fine-tuning.

Study on Density Ratio-rr

The density ratio rr sets the number of coordinates available for perturbation. Table 7 compares the evaluated densities on Qwen2.5-3B.

Ratio-rrARC-EARC-CHellaSwagWinoGrandeOpenBookQA
0.0585.057.871.868.654.0
0.185.460.874.469.752.8
0.283.859.572.069.553.6
0.484.259.871.467.152.8
MeZO82.957.869.966.349.0

Table 7. The effects of density ratio-rr for Fine-tuning Qwen2.5-3B with WIS-MeZO.

WIS-MeZO matches or improves on MeZO at every reported task–density combination. A density of r=0.1r = 0.1 gives the highest score on four of the five tasks in this table, while OpenBookQA performs best at r=0.05r = 0.05.

Effects of Calibration Size and Computation Cost

Table 8 studies calibration-set size on Qwen2.5-3B. Performance improves with larger calibration sets and stabilizes around 128 examples, which we use by default. Table 9 reports the total calibration time. With 128 examples, mask construction takes 0.53–1.26 seconds and is performed once for fixed-mask WIS-MeZO.

Calibration sizeARC-EARC-CHellaSwagWinoGrandeOpenBookQA
3285.060.574.067.454.0
6485.260.574.367.853.0
128 (default)85.460.874.469.754.0
25685.460.574.669.754.0
51285.960.874.569.753.9

Table 8. Effects of the calibration size on WIS-MeZO fine-tuning of Qwen2.5-3B.

Calibration sizeARC-EARC-CHellaSwagWinoGrandeOpenBookQA
320.320.330.430.260.27
640.490.490.720.340.38
128 (default)0.800.891.260.530.60
2561.491.552.360.911.00
5122.702.984.591.661.84

Table 9. Total calibration time (seconds) for Qwen2.5-3B.

Conclusion

We present WIS-MeZO, which uses downstream Wanda scores to select sparse perturbation coordinates for memory-efficient LLM fine-tuning. Our gradient-energy analysis explains the importance of support selection, while experiments across 16 benchmarks and models from 3B to 32B show improved aggregate performance over MeZO. By combining forward-only updates with one-time calibration and compact mask storage, WIS-MeZO also offers a promising direction for resource-constrained on-device adaptation.

Proofs

Notation and Gradient Energy

Table 10 summarizes the notation. The support-selection analysis considers one linear layer. The alignment and convergence results use the total selected dimension across the optimized parameter blocks.

SymbolDescription
W,M,Z\mathbf{W},\mathbf{M},\mathbf{Z}Weights, binary support mask, and Gaussian perturbation.
G,G^ϵ,G^0\mathbf{G},\widehat{\mathbf{G}}_{\epsilon},\widehat{\mathbf{G}}_{0}Stochastic gradient, finite-difference estimate, and its directional limit.
x,δ\mathbf{x},\boldsymbol{\delta}Per-observation input and backpropagated signal.
X∈Rn×B\mathbf{X}\in\mathbb{R}^{n\times B}Matrix of calibration activation observations for one layer.
vj,v^jv_j,\widehat{v}_jPopulation and empirical second moments of feature jj.
Si,jS_{i,j}Wanda score of weight Wi,j\mathbf{W}_{i,j}.
LWIS,i,Ii∗,Ai\mathcal{L}_{\mathrm{WIS},i},\mathcal{I}_i^*,\mathcal{A}_iWIS support, oracle support, and their union in row ii.
r, k=rnr,\ k=rnPerturbation density and selected coordinates per row of a layer.
d=∑ℓmℓnℓd=\sum_{\ell}m_{\ell}n_{\ell}Total number of parameters in the optimized blocks.
s=deff=∑ℓmℓkℓs=d_{\mathrm{eff}}=\sum_{\ell}m_{\ell}k_{\ell}Total number of selected perturbation coordinates.
γ,ν\gamma,\nuBounds on input–error dependence.
K,ξK,\xiNormalized sub-Gaussian constant and calibration error tolerance.
κi,κi,ξ,κξ\kappa_i,\kappa_{i,\xi},\kappa_{\xi}Row-wise comparison factors and their minimum.
ρ,ρ∗,ρmin⁡\rho,\rho^*,\rho_{\min}Captured energy, oracle capture, and a trajectory-wise lower bound.
Ci,low,Ci,highC_{i,\mathrm{low}},C_{i,\mathrm{high}}Gradient–saliency comparison constants.
ϵ,η,τ\epsilon,\eta,\tauPerturbation radius, learning rate, and mask refresh interval.
L,μ,σ2L,\mu,\sigma^2Smoothness, PL, and stochastic-variance constants.
a=s+2, σϵ2a=s+2,\ \sigma_{\epsilon}^2Gaussian moment factor and variance bound including finite differences.
∣⋅∣2,∣⋅∣F|\cdot|_2,|\cdot|_FEuclidean and Frobenius norms.
⟨⋅,⋅⟩,⊙\langle\cdot,\cdot\rangle,\odotFrobenius inner product and element-wise multiplication.

Table 10. Summary of notation.

For a linear layer y=Wx\mathbf{y}=\mathbf{W}\mathbf{x}, the per-example gradient satisfies

Gi,j=∂f∂Wi,j=δixj,δi=∂f∂yi.G_{i,j}=\frac{\partial f}{\partial W_{i,j}}=\delta_i x_j,\qquad\delta_i=\frac{\partial f}{\partial y_i}.

The calibration analysis concerns E[Gi,j2]\mathbb{E}[G_{i,j}^2] over the downstream distribution at a fixed parameter value. For BB calibration observations,

v^j=1B∑b=1B(xj(b))2,vj=E[xj2],Si,j2=BWi,j2v^j.\widehat{v}_j=\frac{1}{B}\sum_{b=1}^{B}\left(x_j^{(b)}\right)^2,\qquad v_j=\mathbb{E}[x_j^2],\qquad S_{i,j}^2=BW_{i,j}^2\widehat{v}_j.

The following proofs establish the energy-capture and convergence results in the main text.

Proof of Lemma 5.3

Proof. Assumption 5.1 gives

γE[δi2]vj≤E[Gi,j2]≤νE[δi2]vj.(2)\gamma\mathbb{E}[\delta_i^2]v_j\leq\mathbb{E}[G_{i,j}^2]\leq\nu\mathbb{E}[\delta_i^2]v_j. \tag*{(2)}

Under Assumption 5.2, the squared activation is sub-exponential with scale bounded by a constant multiple of KvjKv_j. For features with vj>0v_j > 0, concentration of the empirical mean gives, for a universal constant c>0c > 0 and ξ∈(0,1)\xi\in(0,1),

Pr⁡(∣v^j−vj∣≥ξvj)≤2exp⁡(−cBξ2K2).\Pr\left(\lvert\hat{v}_j-v_j\rvert\ge\xi v_j\right) \le2\exp\left(-\frac{cB\xi^2}{K^2}\right).

Features with vj=0v_j=0 satisfy v^j=0\hat{v}_j=0 almost surely. A union bound over the nn features yields

(1−ξ)vj≤v^j≤(1+ξ)vj,j=1,…,n,(1-\xi)v_j \le\hat{v}_j \le(1+\xi)v_j,\qquad j=1,\ldots,n,

with probability at least 1−O(n−1)1-O(n^{-1}) when B=Ω(K2log⁡n/ξ2)B=\Omega(K^2\log n/\xi^2). On this event,

Si,j2B(1+ξ)Wi,j2≤vj≤Si,j2B(1−ξ)Wi,j2.(3)\frac{S_{i,j}^2}{B(1+\xi)W_{i,j}^2} \le v_j \le\frac{S_{i,j}^2}{B(1-\xi)W_{i,j}^2}. \tag*{(3)}

For every nonzero-energy row and j∈Aij\in\mathcal{A}_i, combining Eqs. equation (2) and equation (3) gives

γE[δi,j2]B(1+ξ)wmax⁡,iSi,j2≤E[Gi,j2]≤νE[δi,j2]B(1−ξ)wmin⁡,iSi,j2.\frac{\gamma\mathbb{E}[\delta_{i,j}^2]}{B(1+\xi)w_{\max,i}}S_{i,j}^2 \le\mathbb{E}[G_{i,j}^2] \le\frac{\nu\mathbb{E}[\delta_{i,j}^2]}{B(1-\xi)w_{\min,i}}S_{i,j}^2.

The two coefficients define Ci,lowC_{i,\mathrm{low}} and Ci,highC_{i,\mathrm{high}}, and

Ci,lowCi,high=γwmin⁡,iνwmax⁡,i1−ξ1+ξ=κi,ξ.\frac{C_{i,\mathrm{low}}}{C_{i,\mathrm{high}}} =\frac{\gamma w_{\min,i}}{\nu w_{\max,i}}\frac{1-\xi}{1+\xi} =\kappa_{i,\xi}.

Rows with zero gradient energy contribute zero to the global ratio.

Proof of Theorem 5.4

Proof. Define the row-wise oracle over the full row by

Ii∗∈arg max⁡I⊆{1,…,n}, ∣I∣=k ∑j∈IE[Gi,j2].\mathcal{I}_i^* \in\underset{I\subseteq\{1,\ldots,n\},\,|I|=k}{\operatorname{arg\,max}}\ \sum_{j\in I}\mathbb{E}[G_{i,j}^2].

The WIS support maximizes the sum of squared saliency scores among all sets of size kk. On the concentration event in Lemma 5.3, both supports belong to Ai\mathcal{A}_i, so

∑j∈IWIS,iE[Gi,j2]≥Ci,low∑j∈IWIS,iSi,j2≥Ci,low∑j∈Ii∗Si,j2≥κi,ξ∑j∈Ii∗E[Gi,j2].\begin{aligned} \sum_{j\in\mathcal{I}_{\mathrm{WIS},i}}\mathbb{E}[G_{i,j}^2] &\ge C_{i,\mathrm{low}}\sum_{j\in\mathcal{I}_{\mathrm{WIS},i}}S_{i,j}^2\\ &\ge C_{i,\mathrm{low}}\sum_{j\in\mathcal{I}_i^*}S_{i,j}^2\\ &\ge\kappa_{i,\xi}\sum_{j\in\mathcal{I}_i^*}\mathbb{E}[G_{i,j}^2]. \end{aligned}

Dividing by the total expected energy in the row and summing with weights ωi\omega_i gives

ρWIS=∑iωiρWIS,i≥∑iωiκi,ξρi∗≥κξρ∗.\rho_{\mathrm{WIS}}=\sum_i\omega_i\rho_{\mathrm{WIS},i}\ge\sum_i\omega_i\kappa_{i,\xi}\rho_i^*\ge\kappa_\xi\rho^*.

For a uniform row-wise mask independent of the gradient, each coordinate is included with probability rr. Therefore,

EMrand[ρ(Mrand)]=∑i,jrE[Gi,j2]E[∥G∥F2]=r.\mathbb{E}_{M_{\mathrm{rand}}}\left[\rho(M_{\mathrm{rand}})\right] =\frac{\sum_{i,j}r\mathbb{E}[G_{i,j}^2]}{\mathbb{E}[\lVert G\rVert_F^2]}=r.

The sufficient condition κξρ∗>r\kappa_\xi\rho^*>r then gives the stated comparison.

Proof of Lemma 5.5

Proof. Fix the parameter value and mask. Write U=(Z⊙M)/r\mathbf{U}=(\mathbf{Z}\odot\mathbf{M})/\sqrt{r}, where the entries of Z\mathbf{Z} are independent standard Gaussians, and G^0=⟨G,U⟩U\widehat{\mathbf{G}}_0=\langle\mathbf{G},\mathbf{U}\rangle\mathbf{U}. Taking the perturbation expectation first gives

E[⟨G^0,G⟩]=EB[⟨EZ[G^0∣G],G⟩]=1rEB[∥M⊙G∥F2]=ρrEB[∥G∥F2].(4)\begin{aligned} \mathbb{E}[\langle\widehat{\mathbf{G}}_0,\mathbf{G}\rangle]=\mathbb{E}_{B}[\langle\mathbb{E}_{Z}[\widehat{\mathbf{G}}_0\mid\mathbf{G}],\mathbf{G}\rangle] \\ &=\frac{1}{r}\mathbb{E}_{B}[\|\mathbf{M}\odot\mathbf{G}\|_{F}^{2}] \\ &=\frac{\rho}{r}\mathbb{E}_{B}[\|\mathbf{G}\|_{F}^{2}]. \tag*{(4)} \end{aligned}

The Gaussian fourth-moment identity in the ss selected coordinates yields

EZ[∥G^0∥F2∣G]=s+2r2∥M⊙G∥F2.\mathbb{E}_{Z}[\|\widehat{\mathbf{G}}_0\|_{F}^{2}\mid\mathbf{G}]=\frac{s+2}{r^{2}}\|\mathbf{M}\odot\mathbf{G}\|_{F}^{2}.

Hence,

E[∥G^0∥F2]=s+2r2ρE[∥G∥F2].(5)\mathbb{E}[\|\widehat{\mathbf{G}}_0\|_{F}^{2}]=\frac{s+2}{r^{2}}\rho\mathbb{E}[\|\mathbf{G}\|_{F}^{2}]. \tag*{(5)}

Substituting Eqs. equation 4 and equation (5) into the moment-normalized alignment gives

Align⁡(G^0,G)=ρs+2.\operatorname{Align}(\widehat{\mathbf{G}}_0,\mathbf{G})=\sqrt{\frac{\rho}{s+2}}.

For a uniform random mask independent of this gradient distribution, taking an additional expectation over masks replaces ρ\rho by rr. The corresponding ratio of moment-normalized alignment values is ρ/r\sqrt{\rho/r}.

Proof of Theorem 5.8

Proof. Condition on the fixed calibration mask and the optimization history before iteration tt. Write Et\mathbb{E}_t for expectation over the current mini-batch and perturbation, and define

Ht=M⊙∇f(Wt),U=Z⊙Mr,a=s+2.\mathbf{H}_t=\mathbf{M}\odot\nabla f(\mathbf{W}^{t}),\qquad\mathbf{U}=\frac{\mathbf{Z}\odot\mathbf{M}}{\sqrt{r}},\qquad a=s+2.

The actual central-difference estimator and its directional limit are

G^ϵt=f(Wt+ϵU,Bt)−f(Wt−ϵU,Bt)2ϵU,G^0t=⟨Gt,U⟩U.\widehat{\mathbf{G}}_{\epsilon}^{t}=\frac{f(\mathbf{W}^{t}+\epsilon\mathbf{U},\mathcal{B}^{t})-f(\mathbf{W}^{t}-\epsilon\mathbf{U},\mathcal{B}^{t})}{2\epsilon}\mathbf{U},\qquad\widehat{\mathbf{G}}_{0}^{t}=\langle\mathbf{G}^{t},\mathbf{U}\rangle\mathbf{U}.

Gaussian symmetry and integration by parts give

Et[G^ϵt]=1rM⊙EZ[∇f(Wt+ϵU)].\mathbb{E}_{t}[\widehat{\mathbf{G}}_{\epsilon}^{t}]=\frac{1}{r}\mathbf{M}\odot\mathbb{E}_{Z}[\nabla f(\mathbf{W}^{t}+\epsilon\mathbf{U})].

For bt=Et[G^ϵt]−Ht/r\mathbf{b}_{t}=\mathbb{E}_{t}[\widehat{\mathbf{G}}_{\epsilon}^{t}]-\mathbf{H}_{t}/r, smoothness and E∥U∥F2=s/r\mathbb{E}\|\mathbf{U}\|_{F}^{2}=s/r imply

∥bt∥F2≤1r2EZ[∥M⊙(∇f(Wt+ϵU)−∇f(Wt))∥F2]≤L2ϵ2r2EZ[∥U∥F2]=L2ϵ2sr3.(6)\begin{aligned} \|\mathbf{b}_{t}\|_{F}^{2}\le\frac{1}{r^{2}}\mathbb{E}_{Z}[\|\mathbf{M}\odot(\nabla f(\mathbf{W}^{t}+\epsilon\mathbf{U})-\nabla f(\mathbf{W}^{t}))\|_{F}^{2}] \\ &\le\frac{L^{2}\epsilon^{2}}{r^{2}}\mathbb{E}_{Z}[\|\mathbf{U}\|_{F}^{2}] \\ &=\frac{L^{2}\epsilon^{2}s}{r^{3}}. \tag*{(6)} \end{aligned}

For each mini-batch, the fundamental theorem of calculus and Assumption 5.6 give

∥G^ϵt−G^0t∥F≤∥U∥F2ϵ∫−ϵϵ∣⟨∇f(Wt+vU,Bt)−Gt,U⟩∣ dv≤L∥U∥F32ϵ∫−ϵϵ∣v∣ dv=Lϵ2∥U∥F3.\begin{aligned} \|\widehat{\mathbf{G}}_{\epsilon}^{t}-\widehat{\mathbf{G}}_{0}^{t}\|_{F}\le\frac{\|\mathbf{U}\|_{F}}{2\epsilon}\int_{-\epsilon}^{\epsilon}\left|\langle\nabla f(\mathbf{W}^{t}+v\mathbf{U},\mathcal{B}^{t})-\mathbf{G}^{t},\mathbf{U}\rangle\right|\,dv \\ &\le\frac{L\|\mathbf{U}\|_{F}^{3}}{2\epsilon}\int_{-\epsilon}^{\epsilon}|v|\,dv \\ &=\frac{L\epsilon}{2}\|\mathbf{U}\|_{F}^{3}. \end{aligned}

Since r∥U∥F2r\lVert U\rVert_{F}^{2} has a chi-squared distribution with ss degrees of freedom,

Et[∥G^ϵt−G^0t∥F2]≤L2ϵ24E[∥U∥F6]=L2ϵ2s(s+2)(s+4)4r3.(7)\mathbb{E}_{t}\left[\left\lVert\widehat{G}_{\epsilon}^{t}-\widehat{G}_{0}^{t}\right\rVert_{F}^{2}\right]\leq\frac{L^{2}\epsilon^{2}}{4}\mathbb{E}\left[\lVert U\rVert_{F}^{6}\right]=\frac{L^{2}\epsilon^{2}s(s+2)(s+4)}{4r^{3}}. \tag*{(7)}

By Lemma 5.5 and the conditional variance bound,

Et[∥G^0t∥F2]=ar2Et[∥M⊙Gt∥F2]≤ar2(∥Ht∥F2+σ2).\begin{aligned} \mathbb{E}_{t}\left[\left\lVert\widehat{G}_{0}^{t}\right\rVert_{F}^{2}\right]=\frac{a}{r^{2}}\mathbb{E}_{t}\left[\lVert\mathbf{M}\odot G^{t}\rVert_{F}^{2}\right] \\ &\leq\frac{a}{r^{2}}\left(\lVert H_{t}\rVert_{F}^{2}+\sigma^{2}\right). \end{aligned}

Combining this inequality with Eq. equation 7 and ∥A+B∥F2≤2∥A∥F2+2∥B∥F2\lVert A+B\rVert_{F}^{2}\leq2\lVert A\rVert_{F}^{2}+2\lVert B\rVert_{F}^{2} yields

Et[∥G^ϵt∥F2]≤2ar2(∥Ht∥F2+σϵ2),σϵ2=σ2+L2ϵ2s(s+4)4r.(8)\mathbb{E}_{t}\left[\left\lVert\widehat{G}_{\epsilon}^{t}\right\rVert_{F}^{2}\right]\leq\frac{2a}{r^{2}}\left(\lVert H_{t}\rVert_{F}^{2}+\sigma_{\epsilon}^{2}\right),\qquad\sigma_{\epsilon}^{2}=\sigma^{2}+\frac{L^{2}\epsilon^{2}s(s+4)}{4r}. \tag*{(8)}

Both the estimator and its bias are supported on M\mathbf{M}. Applying Young’s inequality gives

Et[⟨∇f(Wt),G^ϵt⟩]=1r∥Ht∥F2+⟨Ht,bt⟩≥34r∥Ht∥F2−r∥bt∥F2.(9)\begin{aligned} \mathbb{E}_{t}[\langle\nabla f(\mathbf{W}^{t}),\widehat{G}_{\epsilon}^{t}\rangle]=\frac{1}{r}\lVert H_{t}\rVert_{F}^{2}+\langle H_{t},b_{t}\rangle\\ &\geq\frac{3}{4r}\lVert H_{t}\rVert_{F}^{2}-r\lVert b_{t}\rVert_{F}^{2}. \tag*{(9)} \end{aligned}

Using Wt+1=Wt−ηtG^ϵt\mathbf{W}^{t+1}=\mathbf{W}^{t}-\eta_{t}\widehat{G}_{\epsilon}^{t} and population smoothness,

Et[f(Wt+1)]≤f(Wt)−ηtEt[⟨∇f(Wt),G^ϵt⟩]+Lηt22Et[∥G^ϵt∥F2].\mathbb{E}_{t}[f(\mathbf{W}^{t+1})]\leq f(\mathbf{W}^{t})-\eta_{t}\mathbb{E}_{t}[\langle\nabla f(\mathbf{W}^{t}),\widehat{G}_{\epsilon}^{t}\rangle]+\frac{L\eta_{t}^{2}}{2}\mathbb{E}_{t}\left[\left\lVert\widehat{G}_{\epsilon}^{t}\right\rVert_{F}^{2}\right].

For ηt≤r/(4La)\eta_{t}\leq r/(4La), substituting Eqs. equation 6, equation 8, and equation 9 gives

Et[f(Wt+1)]≤f(Wt)−ηt2r∥Ht∥F2+Laηt2r2σϵ2+L2ϵ2sηtr2.(10)\mathbb{E}_{t}[f(\mathbf{W}^{t+1})]\leq f(\mathbf{W}^{t})-\frac{\eta_{t}}{2r}\lVert H_{t}\rVert_{F}^{2}+\frac{La\eta_{t}^{2}}{r^{2}}\sigma_{\epsilon}^{2}+\frac{L^{2}\epsilon^{2}s\eta_{t}}{r^{2}}. \tag*{(10)}

Now set ηt=r/(4Lat+1)\eta_{t}=r/(4La\sqrt{t+1}). Taking full expectations, summing, and using f≥f∗f\geq f^{*} and the deterministic lower bound on captured energy yield

ρmin⁡∑t=0T−1E[∥∇f(Wt)∥F2]t+1≤8LaΔf+σϵ22∑t=0T−11t+1+2L2ϵ2sr∑t=0T−11t+1\rho_{\min}\sum_{t=0}^{T-1}\frac{\mathbb{E}[\lVert\nabla f(\mathbf{W}^{t})\rVert_{F}^{2}]}{\sqrt{t+1}}\leq8La\Delta f+\frac{\sigma_{\epsilon}^{2}}{2}\sum_{t=0}^{T-1}\frac{1}{t+1}+\frac{2L^{2}\epsilon^{2}s}{r}\sum_{t=0}^{T-1}\frac{1}{\sqrt{t+1}}
≤8LaΔf+σϵ22(1+log⁡T)+4L2ϵ2sTr.\leq8La\Delta f+\frac{\sigma_{\epsilon}^{2}}{2}(1+\log T)+\frac{4L^{2}\epsilon^{2}s\sqrt{T}}{r}.

Finally, 1/t+1≥1/T1/\sqrt{t+1}\geq1/\sqrt{T} for t<Tt<T, so

1T∑t=0T−1E[∥∇f(Wt)∥F2]≤8LaΔf+12σϵ2(1+log⁡T)ρmin⁡T+4L2ϵ2srρmin⁡.\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[\lVert\nabla f(\mathbf{W}^{t})\rVert_{F}^{2}]\leq\frac{8La\Delta f+\frac{1}{2}\sigma_{\epsilon}^{2}(1+\log T)}{\rho_{\min}\sqrt{T}}+\frac{4L^{2}\epsilon^{2}s}{r\rho_{\min}}.

Convergence under the PL Condition

The same descent inequality gives geometric convergence to a noise and finite-difference neighborhood under a PL condition.

Assumption A.1 (μ\mu-PL Condition). For 0<μ≤L0<\mu\leq L,

∥∇f(W)∥F2≥2μ(f(W)−f∗).\lVert\nabla f(\mathbf{W})\rVert_{F}^{2}\geq2\mu(f(\mathbf{W})-f^{*}).

Theorem A.2 (Convergence under the PL Condition). Suppose Assumptions 5.6, 5.7, and A.1 hold, together with the fixed-support energy condition in Theorem 5.8. Let a=s+2a=s+2, retain σϵ2\sigma_{\epsilon}^{2} from that theorem, and take η=αr/(4La)\eta=\alpha r/(4La) with α∈(0,1]\alpha\in(0,1]. Then

E[f(WT)−f∗]≤(1−αμρmin⁡4La)TΔf+ασϵ24μρmin⁡+L2ϵs2rμρmin⁡.\mathbb{E}[f(\mathbf{W}^{T})-f^{*}] \leq\left(1-\frac{\alpha\mu\rho_{\min}}{4La}\right)^{T}\Delta f+\frac{\alpha\sigma_{\epsilon}^{2}}{4\mu\rho_{\min}}+\frac{L^{2}\epsilon_{s}^{2}}{r\mu\rho_{\min}}.

Proof. Write Δt=E[f(Wt)−f∗]\Delta_{t}=\mathbb{E}[f(\mathbf{W}^{t})-f^{*}]. Substituting the constant step size into Eq. equation 10 and applying the support-energy and PL conditions gives

Δt+1≤(1−αμρmin⁡4La)Δt+α2σϵ216La+αLϵs24ar.\Delta_{t+1}\leq\left(1-\frac{\alpha\mu\rho_{\min}}{4La}\right)\Delta_{t}+\frac{\alpha^{2}\sigma_{\epsilon}^{2}}{16La}+\frac{\alpha L\epsilon_{s}^{2}}{4ar}.

Set q=1−αμρmin⁡/(4La)∈[0,1)q=1-\alpha\mu\rho_{\min}/(4La)\in[0,1). Iterating this inequality yields

ΔT≤qTΔ0+(α2σϵ216La+αLϵs24ar)∑t=0T−1qt≤qTΔ0+ασϵ24μρmin⁡+L2ϵs2rμρmin⁡.\begin{aligned} \Delta_{T}\leq q^{T}\Delta_{0}+\left(\frac{\alpha^{2}\sigma_{\epsilon}^{2}}{16La}+\frac{\alpha L\epsilon_{s}^{2}}{4ar}\right)\sum_{t=0}^{T-1}q^{t} \\ &\leq q^{T}\Delta_{0}+\frac{\alpha\sigma_{\epsilon}^{2}}{4\mu\rho_{\min}}+\frac{L^{2}\epsilon_{s}^{2}}{r\mu\rho_{\min}}. \end{aligned}

Experimental Details

For the GLUE, SQuAD, and SuperGLUE experiments, all ZO methods train for 20,000 steps with a batch size of 16; FT and LoRA use a batch size of 8. Checkpoints are selected by validation performance. We use a sequence length of 2048, constant learning rates, and zero weight decay for ZO methods, following the settings in MeZO and Sparse-MeZO [23, 21].

Table 11–Table 13 report the hyperparameter search and selected configurations. Results on GLUE and SQuAD are averaged over five seeds, while SuperGLUE results are from one run. WIS-MeZO uses 128 calibration examples from downstream training data and a fixed mask.

MethodsHyperparametersValues
FTBatch size8
FTLearning rate{1.0e−6,5.0e−6,1.0e−5,5.0e−5}\{1.0\mathrm{e}{-6}, 5.0\mathrm{e}{-6}, 1.0\mathrm{e}{-5}, 5.0\mathrm{e}{-5}\}
FT(β1,β2)(\beta_{1}, \beta_{2})(0.9,0.999)(0.9, 0.999)
LoRABatch size8
LoRALearning rate{1.0e−5,5.0e−5,1.0e−4,5.0e−4}\{1.0\mathrm{e}{-5}, 5.0\mathrm{e}{-5}, 1.0\mathrm{e}{-4}, 5.0\mathrm{e}{-4}\}
LoRARank8
LoRAAlpha-α\alpha16
LoRA(β1,β2)(\beta_{1}, \beta_{2})(0.9,0.999)(0.9, 0.999)
MeZOBatch size16
MeZOLearning rate{5.0e−8,1.0e−7,2.5e−7,5.0e−7}\{5.0\mathrm{e}{-8}, 1.0\mathrm{e}{-7}, 2.5\mathrm{e}{-7}, 5.0\mathrm{e}{-7}\}
MeZOϵ\epsilon{1.0e−3,1.0e−4}\{1.0\mathrm{e}{-3}, 1.0\mathrm{e}{-4}\}
LOZOBatch size16
LOZOLearning rate{5.0e−8,1.0e−7,2.5e−7,5.0e−7}\{5.0\mathrm{e}{-8}, 1.0\mathrm{e}{-7}, 2.5\mathrm{e}{-7}, 5.0\mathrm{e}{-7}\}
LOZOϵ\epsilon{1.0e−3,1.0e−4}\{1.0\mathrm{e}{-3}, 1.0\mathrm{e}{-4}\}
LOZORank{1,2}\{1, 2\}
LOZOInterval50
Sparse-MeZOBatch size16
Sparse-MeZOLearning rate{5.0e−7,1.0e−6,2.5e−6,5.0e−6}\{5.0\mathrm{e}{-7}, 1.0\mathrm{e}{-6}, 2.5\mathrm{e}{-6}, 5.0\mathrm{e}{-6}\}
Sparse-MeZOϵ\epsilon{1.0e−3,1.0e−4}\{1.0\mathrm{e}{-3}, 1.0\mathrm{e}{-4}\}
Sparse-MeZODensity ratio-rr{0.05,0.1,0.2,0.4}\{0.05, 0.1, 0.2, 0.4\}
Random-MeZOBatch size16
Random-MeZOLearning rate{5.0e−8,1.0e−7,2.5e−7,5.0e−7}\{5.0\mathrm{e}{-8}, 1.0\mathrm{e}{-7}, 2.5\mathrm{e}{-7}, 5.0\mathrm{e}{-7}\}
Random-MeZOϵ\epsilon{1.0e−3,1.0e−4}\{1.0\mathrm{e}{-3}, 1.0\mathrm{e}{-4}\}
Random-MeZODensity ratio-rr{0.05,0.1,0.2,0.4}\{0.05, 0.1, 0.2, 0.4\}
WIS-MeZOBatch size16
WIS-MeZOLearning rate{5.0e−8,1.0e−7,2.5e−7,5.0e−7}\{5.0\mathrm{e}{-8}, 1.0\mathrm{e}{-7}, 2.5\mathrm{e}{-7}, 5.0\mathrm{e}{-7}\}
WIS-MeZOϵ\epsilon{1.0e−3,1.0e−4}\{1.0\mathrm{e}{-3}, 1.0\mathrm{e}{-4}\}
WIS-MeZODensity ratio-rr{0.05,0.1,0.2,0.4}\{0.05, 0.1, 0.2, 0.4\}
WISD-MeZOBatch size16
WISD-MeZOLearning rate{5.0e−8,1.0e−7,2.5e−7,5.0e−7}\{5.0\mathrm{e}{-8}, 1.0\mathrm{e}{-7}, 2.5\mathrm{e}{-7}, 5.0\mathrm{e}{-7}\}
WISD-MeZOϵ\epsilon{1.0e−3,1.0e−4}\{1.0\mathrm{e}{-3}, 1.0\mathrm{e}{-4}\}
WISD-MeZODensity ratio-rr{0.05,0.1,0.2,0.4}\{0.05, 0.1, 0.2, 0.4\}
WISD-MeZOFrequency-τ\tau{2000,5000,10000}\{2000, 5000, 10000\}

Table 11. The hyperparameter grids used for the experiments in the main text.

ModelsMethodsCBMultiRCWSCReCoRDBoolQ
LLaMA3.2-3BFT1.0e-51.0e-51.0e-51.0e-61.0e-5
LLaMA3.2-3BLoRA5.0e-41.0e-45.0e-41.0e-55.0e-4
LLaMA3.2-3BMeZO2.5e-75.0e-72.5e-72.5e-71.0e-7
LLaMA3.2-3BLOZO1.0e-72.5e-71.0e-72.5e-71.0e-7
LLaMA3.2-3BSparse-MeZO1.0e-6, 0.41.0e-6, 0.25.0e-6, 0.41.0e-6, 0.22.5e-6, 0.2
LLaMA3.2-3BRandom-MeZO5.0e-7, 0.25.0e-7, 0.22.5e-7, 0.15.0e-7, 0.45.0e-7, 0.2
LLaMA3.2-3BWIS-MeZO2.5e-7, 0.12.5e-7, 0.15.0e-7, 0.051.0e-7, 0.21.0e-7, 0.05
LLaMA3.2-3BWISD-MeZO2.5e-7, 0.1, 20002.5e-7, 0.1, 50001.0e-7, 0.05, 20001.0e-7, 0.2, 100001.0e-7, 0.1, 10000
Qwen2.5-3BFT5.0e-51.0e-51.0e-51.0e-61.0e-5
Qwen2.5-3BLoRA5.0e-55.0e-51.0e-51.0e-51.0e-4
Qwen2.5-3BMeZO5.0e-72.5e-72.5e-75.0e-72.5e-7
Qwen2.5-3BLOZO1.0e-71.0e-71.0e-72.5e-71.0e-7
Qwen2.5-3BSparse-MeZO2.5e-6, 0.21.0e-6, 0.42.5e-6, 0.25.0e-6, 0.22.5e-6, 0.2
Qwen2.5-3BRandom-MeZO1.0e-7, 0.15.0e-7, 0.42.5e-7, 0.25.0e-7, 0.21.0e-7, 0.1
Qwen2.5-3BWIS-MeZO1.0e-7, 0.15.0e-8, 0.051.0e-7, 0.21.0e-7, 0.11.0e-7, 0.2
Qwen2.5-3BWISD-MeZO2.5e-7, 0.2, 20002.5e-7, 0.4, 50001.0e-7, 0.2, 100001.0e-7, 0.1, 20001.0e-7, 0.2,10000

Table 13. Selected hyperparameters for the SuperGLUE experiments.

Regarding the computing environment, we use Python 3.10.14 and PyTorch 2.8.0 [26]. All experiments were conducted on NVIDIA H20 GPUs.

Additional Gradient Energy Capture Results

We compare Random, W-small, W-large, and WIS supports at matched density. The supports are selected uniformly at random, by small weight magnitude, by large weight magnitude, and by the Wanda score, respectively. For each task, we evaluate their gradient energy capture on 2,000 gradient snapshots from an auxiliary AdamW training run on LLaMA-3.2-3B. The supports are fixed at initialization, and all strategies are evaluated on the same gradients.

For a support mask M\mathrm{M} and a mini-batch gradient Gt\mathbf{G}^{t}, energy capture is measured by

ρt(M)=∥M⊙Gt∥F2∥Gt∥F2.(11)\rho_{t}(\mathrm{M})=\frac{\lVert\mathrm{M}\odot\mathbf{G}^{t}\rVert_{F}^{2}}{\lVert\mathbf{G}^{t}\rVert_{F}^{2}}. \tag*{(11)}

We report the average over optimization steps. Random masking captures an expected fraction rr of gradient energy. At r=0.05r=0.05, WIS preserves over 40% of the energy and exceeds the other selection rules. W-small also outperforms W-large and random masking.

Figure 3–Figure 5 report the additional results at density ratios of 0.1, 0.2, and 0.4, respectively. WIS captures the most gradient energy in nearly all reported task–density settings. Its advantage is largest at low densities. At r=0.4r=0.4 on MNLI, WIS and W-small have comparable capture ratios. W-small consistently captures more energy than W-large and generally exceeds random masking. These comparisons describe how the selected coordinates align with the shared AdamW gradient snapshots and show the contribution of downstream activation information to support selection.

Figure 3. Gradient energy capture ratio ρ\rho on LLaMA-3.2-3B at a density ratio of r=0.1r = 0.1. Random denotes uniform random masking. W-small and W-large select coordinates with the smallest and largest weight magnitudes, respectively, while WIS selects coordinates using Wanda weight–activation scores.

Figure 5. Gradient energy capture ratio ρ\rho on LLaMA-3.2-3B at a density ratio of r=0.4r = 0.4. Random denotes uniform random masking. W-small and W-large select coordinates with the smallest and largest weight magnitudes, respectively, while WIS uses Wanda weight–activation scores.

Detailed Pseudocode

This section provides the detailed implementations of the parameter perturbation and WIS mask construction routines used in Algorithm 1. For each linear layer, W∈Rm×nW \in\mathbb{R}^{m \times n} denotes its weight matrix, Z∈Rm×nZ \in\mathbb{R}^{m \times n} denotes the random perturbation, and M∈{0,1}m×nM \in\{0, 1\}^{m \times n} denotes the corresponding binary mask.

Seed replay regenerates the perturbation for the two loss evaluations and the update, retaining the memory advantages of MeZO.

Algorithm 3 applies the Wanda score and selects the top-kk coordinates in each row. WIS-MeZO constructs the mask once, while WISD-MeZO refreshes it every τ\tau steps.

ModelsMethodsMNLIRTEQQPMRPCSST-2SQuAD
LLaMA3.2-3BFT1.0e-51.0e-51.0e-51.0e-51.0e-51.0e-5
LLaMA3.2-3BLoRA5.0e-45.0e-45.0e-45.0e-45.0e-45.0e-5
LLaMA3.2-3BMeZO5.0e-72.5e-75.0e-72.5e-75.0e-72.5e-7
LLaMA3.2-3BLOZO1.0e-72.5e-71.0e-72.5e-72.5e-72.5e-7
LLaMA3.2-3BSparse-MeZO2.5e-6, 0.21.0e-6, 0.21.0e-6, 0.41.0e-6, 0.41.0e-6, 0.41.0e-6, 0.2
LLaMA3.2-3BRandom-MeZO5.0e-7, 0.25.0e-7, 0.25.0e-7, 0.45.0e-7, 0.25.0e-7, 0.25.0e-7, 0.2
LLaMA3.2-3BWIS-MeZO2.5e-7, 0.22.5e-7, 0.25.0e-7, 0.42.5e-7, 0.11.0e-7, 0.11.0e-7, 0.1
LLaMA3.2-3BWISD-MeZO2.5e-7, 0.2, 50002.5e-7, 0.1, 20002.5e-7, 0.2, 100002.5e-7, 0.1, 20001.0e-7, 0.1, 50001.0e-7, 0.1, 5000
Qwen2.5-3BFT1.0e-51.0e-51.0e-51.0e-51.0e-51.0e-5
Qwen2.5-3BLoRA1.0e-41.0e-45.0e-51.0e-41.0e-55.0e-4
Qwen2.5-3BMeZO1.0e-71.0e-72.5e-75.0e-75.0e-75.0e-7
Qwen2.5-3BLOZO5.0e-81.0e-71.0e-71.0e-72.5e-72.5e-7
Qwen2.5-3BSparse-MeZO2.5e-6, 0.12.5e-6, 0.45.0e-6, 0.15.0e-6, 0.25.0e-6, 0.45.0e-6, 0.1
Qwen2.5-3BRandom-MeZO1.0e-7, 0.15.0e-7, 0.41.0e-7, 0.12.5e-7, 0.25.0e-7, 0.45.0e-7, 0.4
Qwen2.5-3BWIS-MeZO1.0e-7, 0.21.0e-7, 0.15.0e-8, 0.45.0e-8, 0.22.5e-7, 0.45.0e-8, 0.05
Qwen2.5-3BWISD-MeZO5.0e-8, 0.2, 50005.0e-8, 0.05, 100005.0e-8, 0.4, 20005.0e-8, 0.2, 50002.5e-7, 0.2, 20005.0e-8, 0.05, 5000

Table 12. Selected hyperparameters for the GLUE and SQuAD experiments.

Figure 4. Gradient energy capture ratio ρ\rho on LLaMA-3.2-3B at a density ratio of r=0.2r = 0.2. All mask construction strategies use the same row-wise perturbation density, model parameters, gradient snapshots, and evaluation data.

Algorithm 2 Perturb Parameters

  1. Function PerturbParameters(W,r,ϵ,M,srng\mathbf{W}, r, \epsilon, \mathbf{M}, s_{\mathrm{rng}}):

  2. Input: Weights W∈Rm×n\mathbf{W} \in\mathbb{R}^{m \times n}, density ratio rr, perturbation scale ϵ\epsilon, mask M∈{0,1}m×n\mathbf{M} \in\{0,1\}^{m \times n}, and random seed srngs_{\mathrm{rng}}

  3. Reset the random number generator with seed srngs_{\mathrm{rng}}

  4. Sample Z∈Rm×n\mathbf{Z} \in\mathbb{R}^{m \times n} with i.i.d. standard Gaussian entries

  5. W←W+ϵrZ⊙M\mathbf{W} \leftarrow\mathbf{W} + \frac{\epsilon}{\sqrt{r}}\mathbf{Z} \odot\mathbf{M} {In-place perturbation}

  6. return W\mathbf{W}

Algorithm 3 Compute Mask

  1. Function ComputeMask(W,X,r\mathbf{W}, \mathbf{X}, r):

  2. Input: Weights W∈Rm×n\mathbf{W} \in\mathbb{R}^{m \times n}, calibration activations X∈Rn×B\mathbf{X} \in\mathbb{R}^{n \times B}, and density ratio rr

  3. qj←∥Xj,:∥2q_j \leftarrow\lVert\mathbf{X}_{j,:}\rVert_2, for j=1,…,nj = 1,\ldots,n

  4. Si,j←∣Wi,j∣qjS_{i,j} \leftarrow\lvert W_{i,j}\rvert q_j

  5. k←⌊nr⌋k \leftarrow\lfloor nr \rfloor {Selected parameters per row}

  6. for each row ii in SS do

  7. Ii←TopK⁡(Si,:,k)\mathcal{I}_i \leftarrow\operatorname{TopK}(S_{i,:}, k)

  8. Mi,j←I[j∈Ii]M_{i,j} \leftarrow\mathbb{I}[j \in\mathcal{I}_i], for j=1,…,nj = 1,\ldots,n

  9. end for

  10. return M\mathbf{M}

References

  1. [1]HanQin Cai, Daniel McKenzie, Wotao Yin, and Zhenliang Zhang. Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization, 32(2): 687–714, 2022.DOI
  2. [2]Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. arXiv preprint arXiv:2310.02025, 2023.arxiv.org/abs/2310.02025
  3. [3]Xiangyi Chen, Sijia Liu, Kaidi Xu, Xingguo Li, Xue Lin, Mingyi Hong, and David Cox. Zo-adamm: Zeroth-order adaptive momentum method for black-box optimization. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1910.06513
  4. [4]Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.07698
  5. [5]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  6. [6]Sizhe Dang, yangyangGuo, Yanjun Zhao, Haishan Ye, xiaodong zheng, Guang Dai, and Ivor Tsang. FZOO: Fast zeroth-order optimizer for fine-tuning large language models towards adam-scale speed. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2506.09034
  7. [7]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318–30332, 2022.
  8. [8]Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pages 2943–2952. PMLR, 2020.arxiv.org/abs/1911.11134
  9. [9]Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR, 2023.arxiv.org/abs/2301.00774
  10. [10]Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080, 2024.arxiv.org/abs/2404.08080
  11. [11]Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341–2368, 2013.arxiv.org/abs/1309.5549
  12. [12]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  13. [13]Jiaqi Gu, Chenghao Feng, Zheng Zhao, Zhoufeng Ying, Ray T Chen, and David Z Pan. Efficient on-chip learning for optical neural networks through power-aware sparse zeroth-order optimization. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 7583–7591, 2021.arxiv.org/abs/2012.11148
  14. [14]Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhou Xu. Zeroth-order fine-tuning of LLMs with transferable static sparsity. In The Thirteenth International Conference on Learning Representations, 2025.
  15. [15]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  16. [16]Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pages 2137–2146. PMLR, 2018.arxiv.org/abs/1804.08598
  17. [17]Taejong Joo, Wenhan Xia, Cheolmin Kim, Ming Zhang, and Eugene Ie. On surprising effectiveness of masking updates in adaptive optimizers. arXiv preprint arXiv:2602.15322, 2026.arxiv.org/abs/2602.15322
  18. [18]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.arxiv.org/abs/1412.6980
  19. [19]Wei Lin, Yining Jiang, Qingyu Song, Qiao Xiang, and Hong Xu. Agzo: Activation-guided zeroth-order optimization for llm fine-tuning. arXiv preprint arXiv:2601.17261, 2026.arxiv.org/abs/2601.17261
  20. [20]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.09353
  21. [21]Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse meZO: Less parameters for better performance in zeroth-order LLM fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2402.15751
  22. [22]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  23. [23]Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:50338–50375, 2023.arxiv.org/abs/2305.17333
  24. [24]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391, 2018.
  25. [25]Mayumi Ohta, Nathaniel Berger, Artem Sokolov, and Stefan Riezler. Sparse perturbations for improved convergence in stochastic zeroth-order optimization. In International Conference on Machine Learning, Optimization, and Data Science, pages 39–64. Springer, 2020.arxiv.org/abs/2006.01759
  26. [26]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1912.01703
  27. [27]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
  28. [28]James C Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332–341, 2002.
  29. [29]Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2306.11695
  30. [30]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pages 353–355, 2018.
  31. [31]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1905.00537
  32. [32]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Daiyiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Minfeng Xue, Pei Zhang, Qin Zhu, Rui Meng, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.arxiv.org/abs/2412.15115
  33. [33]Ziming Yu, Pan Zhou, Sike Wang, Jia Li, Mi Tian, and Hua Huang. Zeroth-order fine-tuning of llms in random subspaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4475–4485, 2025.arxiv.org/abs/2410.08989
  34. [34]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019.arxiv.org/abs/1905.07830
  35. [35]Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: a benchmark. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2402.11592
  36. [36]Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, and Zheng Zhang. Mazo: Masked zeroth-order optimization for multi-task fine-tuning of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18537–18554, 2025.
  37. [37]Hanzhen Zhao, Shihong Ding, Cong Fang, and Zhouchen Lin. Pazo: Preconditioned accelerated zeroth-order optimization for fine-tuning llms. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.DOI
  38. [38]Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024.arxiv.org/abs/2403.03507
  39. [39]Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. APOLLO: SGD-like memory, adamw-level performance. In Eighth Conference on Machine Learning and Systems, 2025.arxiv.org/abs/2412.05270

Paper details

vlnc:W_4s36dcxt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents