WorksComputer Science
Breaking Information Islands in Sparse Tuning via Small-World Connectivity
Abstract
Sparse tuning is widely used to adapt large language models due to its parameter efficiency. However, its effectiveness depends not only on the number of trainable parameters, but also on the feature-interaction topology of the sparse update's structural support. Limited or uneven connectivity in this support may isolate subsets of feature dimensions, forming information islands that restrict direct task-specific interactions through the trainable update. To address this issue, we propose Small-World Induced Fine-Tuning (SWIFT), a sparse tuning framework inspired by small-world connectivity. SWIFT combines block-diagonal sparse updates for local feature mixing with a fixed deterministic scattering permutation that routes the mixed outputs across feature groups without additional trainable parameters. We further show theoretically that certain random sparse-update models and block-diagonal updates can exhibit information islands, whereas a sufficiently scattering permutation guarantees positive block-level expansion when the trainable block-diagonal factor consists of equal-sized, structurally dense square blocks. Extensive experiments on commonsense reasoning, natural language generation, and image classification demonstrate that SWIFT outperforms the evaluated parameter-efficient fine-tuning (PEFT) baselines on average and can match or surpass full fine-tuning. Implementation is available at https://github.com/xianchaoguan/SWIFT
Introduction
Large language models (LLMs) have shown strong capabilities across language understanding and generation tasks [1, 2, 3, 4], accelerating the practical deployment of modern AI systems [5, 6]. A common adaptation paradigm is to pre-train on large corpora and then fine-tune the model for specific tasks or domains [7, 8]. However, as model scale increases, full fine-tuning becomes costly in computation, memory, and deployment, limiting its scalability in practice [9]. This has motivated parameter-efficient fine-tuning (PEFT), which aims to retain the adaptation ability of full fine-tuning while substantially reducing training and deployment costs [10, 11, 12].
PEFT methods adapt large models by updating only a small set of parameters [13]. Low-Rank Adaptation (LoRA) [14] is a representative approach, parameterizing weight updates with low-rank matrices while freezing pre-trained weights. Its variants further improve expressiveness, initialization, and optimization behavior [15, 16, 17, 18]. However, low-rank methods are inherently constrained by rank bottlenecks, which may limit expressive capacity and downstream performance [19, 20]. Sparse tuning offers an alternative by directly updating a discrete subset of pretrained weights, maintaining parameter efficiency without imposing an explicit low-rank factorization [21, 22, 23]. Recent structured update parameterizations further suggest that the update structure itself is crucial for efficient adaptation [24, 25]. Yet how such structures shape information propagation across feature dimensions remains insufficiently understood.
From the perspective of forward computation, neural network inference can be viewed as information propagation in a high-dimensional feature space [26]. Full fine-tuning allows dense weight updates, enabling direct task-specific interactions across feature dimensions, as illustrated in Figure 1(a). Sparse tuning, in contrast, updates only a subset of entries or structured components, inducing a sparse feature-interaction graph defined by the update’s structurally trainable support. Limited or uneven connectivity in this graph may leave subsets of feature dimensions weakly connected or isolated, forming information islands that restrict direct task-specific interactions through the trainable update. Such islands arise in purely block-diagonal methods such as DiaBlo [25], which preserve local interactions but lack trainable cross-block communication (Figure 1(c)). Irregular or random sparse methods such as SpaRTA [27] may also exhibit uneven feature-group coverage depending on the sampled support (Figure 1(b)).
Figure 1. Schematic feature-interaction topologies of trainable updates in full fine-tuning, SpaRTA, DiaBlo, and SWIFT. SWIFT combines local feature mixing with permutation-based routing across feature groups.
This observation suggests that the effectiveness of sparse tuning depends not only on the number of trainable parameters, but also on how well the update supports global communication across feature dimensions. Small-world connectivity provides a natural principle for this goal, combining strong local interactions with sparse long-range shortcuts to enable efficient information exchange without dense connectivity [28, 29]. In this context, an effective sparse update should combine local feature mixing with reliable access to distant feature groups, motivating designs that integrate structured local mixing with distributed long-range routing.
In this paper, we propose Small-World Induced Fine-Tuning (SWIFT), a sparse tuning framework inspired by small-world connectivity (Figure 1(d)). SWIFT uses block-diagonal sparse updates for local feature mixing and applies a fixed deterministic scattering permutation to route the mixed outputs across feature groups without additional trainable parameters. Theoretically, we show that certain random sparse-update models and block-diagonal updates can exhibit information islands, whereas a sufficiently scattering permutation guarantees positive block-level expansion when the trainable block-diagonal factor consists of equal-sized, structurally dense square blocks. Extensive experiments on commonsense reasoning, natural language generation, and image classification demonstrate that SWIFT outperforms the evaluated PEFT baselines on average and can match or surpass full fine-tuning.
Related Work
Low-Rank Adaptation
Low-rank adaptation is a widely used PEFT paradigm due to its simplicity and strong empirical performance. LoRA [14] enables efficient adaptation by injecting trainable low-rank matrices while freezing pre-trained weights. Subsequent work improves its expressiveness and optimization through better parameterization, initialization, and architectural design, including DoRA [16], PiSSA [17], HydraLoRA [30], and GOAT [31]. However, low-rank factorization restricts the rank of the parameterized update, which may limit the ability to model complex cross-dimensional interactions as adaptation demands increase [32, 19].
Sparse Fine-Tuning
Sparse fine-tuning directly updates a subset of pre-trained weights, maintaining parameter efficiency without imposing an explicit low-rank factorization [21, 22, 23]. Existing methods differ in how the sparse update is constructed. Data-driven approaches select task-relevant parameters based on importance signals, such as SMT [33] and SpIEL [34]. In contrast, SpaRTA [27] randomly selects a fixed subset of parameters for adaptation. S²FT [22] selects attention heads and FFN channels, then co-permutes coupled weight matrices to organize the selected components into dense submatrices for efficient updates. Related structured parameterizations further highlight the importance of update design: MoRe [24] uses two trainable block-diagonal factors and two fixed permutations in a Monarch-style parameterization, while DiaBlo [25] directly updates dense diagonal blocks.
SWIFT combines a single trainable block-diagonal factor with a fixed deterministic scattering permutation, yielding . This composition routes locally mixed outputs across feature groups without additional trainable parameters. Alongside this update design, we analyze how an explicit scattering condition controls remote-group coverage and expansion under stated structural assumptions.
Method
We present SWIFT, which combines block-diagonal updates with a fixed deterministic scattering permutation to route locally mixed outputs across feature groups. We then introduce Minimum Remote Group Coverage (MRGC), a method-agnostic measure of connectivity in the structural support of trainable updates, focusing on worst-case access to remote groups. Finally, we provide a graph-theoretic analysis of information islands and establish coverage and expansion guarantees under explicit structural and scattering assumptions.
SWIFT Framework
SWIFT is applied to standard linear modules in Transformers, such as query/key/value projections and feed-forward network layers. Consider a frozen pre-trained linear weight in such a module. Given an input activation , the original computation is
SWIFT introduces a sparse trainable update that combines local feature mixing with routing across feature groups. Specifically, we parameterize the effective update as a block-diagonal trainable matrix followed by a fixed permutation:
where is trainable and block diagonal, and is fixed throughout fine-tuning. Within each adapted linear module, only is trainable. We initialize to zero, preserving the original module’s output at initialization. The block structure of provides local feature mixing, while routes the mixed outputs across feature groups. Thus, the effective update introduces cross-group interactions without increasing the number of trainable parameters, as illustrated in Figure 2.
Figure 2. Overview of the SWIFT framework. SWIFT combines trainable block-diagonal updates for local feature mixing with a fixed deterministic scattering permutation that routes the mixed outputs across feature groups. The pre-trained weights remain frozen.
Grouping: block-diagonal sparse updates. SWIFT first partitions the input and output feature dimensions of a linear module into groups. When divides both and , let and . We divide into blocks and enforce the block-diagonal structure
During fine-tuning, all entries within the diagonal blocks are trainable, while the off-diagonal blocks remain zero. The number of trainable parameters is . This structure concentrates the trainable budget on dense within-group interactions, providing efficient local adaptation. By itself, a block-diagonal update has no trainable interactions between differently indexed input and output groups.
Permutation: long-range routing. To enable cross-group interactions without introducing additional trainable parameters, SWIFT composes the block-diagonal update with a fixed permutation . Right multiplication by reorders the output coordinates of , dispersing each block’s mixed outputs across feature groups. Consequently, the effective update connects input groups to dispersed output coordinates under the same parameter budget. The permutation is determined by the layer dimensions and block number, remains fixed throughout training, and requires no optimizer state.
Efficient implementation. In practice, is implemented by index reordering and is never explicitly materialized as a dense matrix. We generate its index array using a reshape–transpose–reshape procedure: output indices are reshaped into an grid, transposed, and flattened. The resulting array is used to gather output coordinates. Writing for the pre-permutation coordinate selected for output coordinate , our convention is
When divides the output width, the number of coordinates routed from any one intermediate output group to any one final output group is either or . Each group therefore reaches exactly target groups. Exact uniform scattering requires , equivalently . Otherwise, the pairwise loads differ by at most one coordinate, with realized scattering bound
We use in all main experiments, for which the input and output widths of all adapted projections are divisible by . For incompatible dimensions, the implementation pads the input and block dimensions to the next multiples of , applies the permutation at the padded output width, and retains the first output coordinates. In this case, coverage and scattering are evaluated on the retained support after slicing.
Global Connectivity and Remote Group Coverage Analysis
The connectivity induced by the effective update is key to understanding sparse fine-tuning. We analyze its structural support: the set of positions at which the update is allowed to vary, including trainable entries initialized to zero. This support determines the possible direct task-specific interactions through the trainable update. A useful measure should therefore capture not only sparsity, but also whether each feature dimension has sufficient access to remote feature groups.
To this end, we introduce Minimum Remote Group Coverage (MRGC), a feature-wise worst-case measure of remote-group connectivity. Consider a square linear layer adapted as , where is frozen and is the effective update. Let denote the structural support of . Each pair represents a possible direct trainable interaction from input feature dimension to output feature dimension .
Let be partitioned into groups, , and let denote the index of the group containing feature dimension . For each source feature dimension , we define the set of directly reachable remote target groups as
The MRGC score is defined as
MRGC measures the worst-case fraction of remote feature groups that each feature dimension can directly reach through the update. Poorly covered features may become bottlenecks for direct cross-group interactions. By excluding within-group interactions, MRGC does not reward purely local updates and, under a shared partition and matched density, provides a method-agnostic measure of remote-group coverage.
At and , SWIFT attains an MRGC of 1.00 at density 1/64, compared with 0.00 for block-diagonal tuning and a mean of 0.41 over 1,000 random masks at the same expected density; dense full fine-tuning also attains 1.00. Appendix C.2 provides the simulation setup and the corresponding group-coverage definition for rectangular layers.
Theory of Small-World Induced Fine-Tuning
We provide a graph-theoretic analysis of connectivity in the structural support of trainable updates. For square layers, we identify input and output coordinates with a common vertex set. Under this view, information islands are subsets of feature dimensions with no outgoing trainable interactions to their complement. We analyze how local mixing and permutation-based routing affect this structural connectivity.
Definition 3.1 (Operator-Induced Interaction Graph). Let be a trainable update with structural support . Its operator-induced interaction graph is the directed graph with and
Under the row-vector convention , an edge represents a possible direct trainable interaction from input coordinate to output coordinate .
To quantify bottlenecks in this support graph, we use directed out-vertex expansion.
Definition 3.2 (Directed Out-Vertex Expansion). For a directed graph and a subset , define
The coordinate-level out-vertex expansion is
When , there exists a nonempty subset of at most half the coordinates with no outgoing edges to its complement in the update-support graph.
Theorem 3.3 (Connectivity Collapse under Independent Random Sparsification). Fix a constant . Consider a sequence of random structural supports on vertices whose outgoing supports are sampled independently across vertices and whose expected average out-degree tends to . Assume that
Then the induced graph contains zero-out-degree vertices with probability approaching , and
Theorem 3.3 identifies a failure mode of random sparse updates in the constant-expected-degree regime. For example, independent Bernoulli sampling with entry probability satisfies the assumption because the zero-out-degree probability is . A zero-out-degree vertex forms a singleton with no outgoing boundary. The fixed-density MRGC simulation above examines uneven group coverage at a different sparsity scale. A detailed proof is provided in Appendix § 9.§ 2.
Theorem 3.4 (Zero Expansion of Block-Diagonal Updates). Let have block-diagonal structural support with equal-sized blocks. Then its induced graph satisfies
Theorem 3.4 formalizes the connectivity bottleneck of purely block-diagonal updates. Each block has no outgoing edges to other blocks, and its size is at most . Thus, a block forms a subset with zero outgoing boundary. A detailed proof is provided in Appendix § 9.§ 3.
SWIFT addresses block isolation by composing with a fixed permutation , which disperses locally mixed outputs across feature blocks. We capture this effect through a scattering condition.
Definition 3.5 (-Uniform Scattering Permutation). Partition into equal-sized blocks with block size . A permutation is -uniformly scattering with respect to this partition if
We use the permutation-matrix convention , so . Smaller indicates more evenly dispersed block overlaps. Exact uniform scattering has and requires .
To characterize expansion across complete feature blocks, write and define the block-level out-vertex expansion as
Because block unions form a restricted family of coordinate subsets,
MRGC measures per-vertex remote-group coverage and does not account for neighborhood overlap across vertices, so it has no general ordering with either expansion measure.
Theorem 3.6 (Expansion of Small-World Induced Fine-Tuning). Let have equal-sized, structurally dense diagonal blocks of size , with
Let correspond to an -uniformly scattering permutation with respect to the same partition. If , then the structural support of the SWIFT update satisfies
The same scattering condition separately yields MRGC and coordinate-level expansion bounds; their statements and proofs are provided in Appendix § 9.§ 1 and Appendix § 9.§ 4. Under exact uniform scattering, , the block-level lower bound is .
Taken together, Theorems 3.3–3.6 identify distinct connectivity properties of sparse update supports and support combining local feature mixing with distributed cross-group routing under a sparse parameter budget. The expansion guarantees apply under the stated assumptions for square layers; rectangular projections are characterized through bipartite group coverage.
Experiments and Results
Baselines
We compare SWIFT with full fine-tuning; low-rank methods LoRA [14], DoRA [16], PiSSA [17], HydraLoRA [30], and GOAT [31]; and sparse or structured methods SpaRTA [27], S²FT [22], SMT [33], SpIEL [34], MoRe [24], and DiaBlo [25]. Full fine-tuning updates all pre-trained parameters. Baseline descriptions and task-specific settings are provided in Appendices § 10.§ 1 and § 8.§ 2.
Datasets
We evaluate SWIFT across 17 datasets in three task categories. For commonsense reasoning, we fine-tune LLaMA3-8B [35] on Commonsense-170K and evaluate on eight datasets [36]. For natural language generation, we separately fine-tune LLaMA2-7B [37] on 100K examples each from MetaMathQA [38] and CodeFeedback [39], using the data prepared by PiSSA [17], and evaluate on GSM8K, MATH, HumanEval, and MBPP. For image classification, we fine-tune CLIP ViT-B/32 [40] on five datasets [31]. Dataset descriptions, references, and experimental settings are provided in Appendices § 8.§ 1 and § 8.§ 2.
Comparative Results
We compare SWIFT with full fine-tuning, LoRA variants, and sparse or structured fine-tuning methods across the three task categories. In Table 1–Table 3, Avg. denotes the arithmetic mean of the displayed task scores. Average-score differences reported below are calculated from the displayed averages.
| Method | Param. (%) | BoolQ | PIQA | SIQA | HellaS. | WinoG. | ARC-e | ARC-c | OBQA | Avg. |
| Full Fine-Tuning | 100 | 73.90 | 86.20 | 79.10 | 93.10 | 85.80 | 88.10 | 78.20 | 84.00 | 83.55 |
| Low-rank adaptation | ||||||||||
| LoRA | 0.70 | 74.58 | 88.41 | 80.81 | 95.31 | 85.87 | 89.60 | 80.12 | 85.40 | 85.01 |
| DoRA | 0.71 | <u>76.09</u> | 88.30 | 80.60 | 95.23 | 86.67 | 90.40 | 80.29 | 88.20 | 85.72 |
| PiSSA | 0.70 | 69.70 | 87.20 | 79.50 | 90.30 | 81.00 | 82.90 | 75.50 | 77.80 | 80.49 |
| HydraLoRA | 0.53 | 72.20 | 85.00 | 79.80 | 89.60 | 87.60 | 79.40 | 74.00 | 80.40 | 81.00 |
| Sparse and structured adaptation | ||||||||||
| SpIEL | 0.70 | 72.10 | 83.60 | 80.00 | 91.80 | 85.40 | 91.20 | 76.80 | 80.80 | 82.71 |
| S²FT | 0.70 | 75.00 | 89.00 | 80.70 | 96.50 | 88.00 | 92.50 | <u>83.40</u> | 87.80 | 86.61 |
| MoRe | 0.70 | 71.93 | 89.28 | 82.24 | 95.22 | 88.79 | <u>93.27</u> | 83.19 | 88.80 | 86.59 |
| SMT | 3.01 | 75.10 | <u>89.90</u> | 82.40 | <u>96.30</u> | 88.80 | 92.60 | 82.80 | <u>89.60</u> | 87.19 |
| SpaRTA | 0.44 | 75.60 | 90.32 | 82.91 | 95.06 | 89.03 | 93.18 | 82.94 | 89.20 | <u>87.28</u> |
| DiaBlo | 0.88 | 74.70 | 89.20 | 82.70 | 95.20 | <u>89.20</u> | 93.10 | 82.90 | <u>89.00</u> | 87.00 |
| SWIFT | 0.88 | 76.30 | 90.32 | 83.31 | 95.90 | 89.27 | 93.52 | 84.18 | 91.60 | 88.05 |
Table 1. Performance comparison of LLaMA3-8B on eight commonsense reasoning datasets. Task scores are accuracies (%). The best and second-best scores in each column are indicated in bold and underlined, respectively, including ties.
| Method | Param. (%) | Cars | DTD | GTSRB | RESISC45 | SVHN | Avg. |
| Full Fine-Tuning | 100 | 60.33 | 73.88 | 98.30 | 93.65 | 96.78 | 84.59 |
| Low-rank adaptation | |||||||
| LoRA | 1.49 | 41.02 | 70.15 | 96.51 | 90.38 | 95.39 | 78.69 |
| DoRA | 1.49 | 40.75 | 71.91 | 97.71 | 90.19 | 95.46 | 79.20 |
| PiSSA | 1.49 | 40.41 | 69.62 | 95.84 | 90.58 | 95.84 | 78.46 |
| GOAT | 2.24 | 51.44 | 73.99 | 98.06 | 93.50 | 96.55 | 82.71 |
| Sparse and structured adaptation | |||||||
| MoRe | 1.68 | 62.63 | 74.26 | 98.16 | 93.68 | 96.24 | 84.99 |
| SpaRTA | 1.60 | 58.40 | 70.40 | 97.90 | 92.90 | 96.40 | 83.20 |
| DiaBlo | 1.70 | 63.67 | 73.67 | 97.98 | 93.56 | 96.49 | 85.07 |
| SWIFT | 1.70 | 67.69 | 76.06 | 98.57 | 94.46 | 97.10 | 86.78 |
Table 3. Performance comparison of CLIP ViT-B/32 on five image classification datasets. Task scores are accuracies (%). Bold and underlined entries indicate the best and second-best scores, respectively.
Commonsense Reasoning
As shown in Table 1, SWIFT achieves the highest average accuracy of 88.05% at a reported trainable-parameter ratio of 0.88%. It outperforms LoRA by 3.04 percentage points and the second-best method, SpaRTA, by 0.77 percentage points. SWIFT also surpasses DiaBlo by 1.05 percentage points, with both methods reporting a trainable-parameter ratio of 0.88%. This comparison supports the benefit of combining block-diagonal adaptation with permutation-based routing.
Natural Language Generation
As shown in Table 2, SWIFT achieves the highest average score of 32.56% at a reported trainable-parameter ratio of 1.50%, surpassing full fine-tuning by 4.26 percentage points. It outperforms LoRA by 7.20 percentage points, the second-best method MoRe by 0.59 percentage points, and SpaRTA by 1.30 percentage points. SWIFT also exceeds DiaBlo by 1.03 percentage points, with both methods reporting a trainable-parameter ratio of 1.50%. SWIFT achieves the best results on GSM8K, MATH, and MBPP, demonstrating effective adaptation across mathematical reasoning and code generation.
| Method | Param. (%) | GSM8K | MATH | HumanEval | MBPP | Avg. |
| Full Fine-Tuning | 100 | 49.05 | 7.22 | 21.34 | 35.59 | 28.30 |
| Low-rank adaptation | ||||||
| LoRA | 4.75 | 42.30 | 5.50 | 18.29 | 35.34 | 25.36 |
| DoRA | 4.76 | 50.42 | 7.26 | 23.78 | 38.85 | 30.08 |
| PiSSA | 4.75 | 53.07 | 7.44 | 21.95 | 37.09 | 29.89 |
| HydraLoRA | 1.87 | 49.58 | 7.02 | 23.17 | 37.34 | 29.28 |
| Sparse and structured adaptation | ||||||
| MoRe | 2.37 | <u>53.82</u> | <u>9.40</u> | <u>26.83</u> | 37.84 | <u>31.97</u> |
| SpaRTA | 0.76 | 53.50 | 8.00 | 27.44 | 36.09 | 31.26 |
| DiaBlo | 1.50 | 53.07 | 9.22 | 26.22 | 37.59 | 31.53 |
| SWIFT | 1.50 | 54.36 | 9.70 | <u>26.83</u> | 39.35 | 32.56 |
Table 2. Performance comparison of LLaMA2-7B on four natural language generation datasets. Scores are percentages. Bold and underlined entries indicate the best and second-best scores, respectively, including ties.
Image Classification
To evaluate applicability beyond language tasks, we test SWIFT on five image classification datasets using CLIP ViT-B/32, as shown in Table 3. SWIFT achieves the highest average accuracy of 86.78%, outperforming full fine-tuning by 2.19 percentage points at a reported trainable-parameter ratio of 1.70%. It also surpasses LoRA by 8.09 percentage points, SpaRTA by 3.58 percentage points, and DiaBlo by 1.71 percentage points. SWIFT achieves the best accuracy on all five evaluated datasets, supporting the effectiveness of its update structure for visual adaptation.
Ablation Study
Update-density sensitivity. We study the sensitivity of SWIFT to the update density by varying the number of blocks in the block-diagonal update. When the input and output dimensions are divisible by , the structural density of each adapted matrix is . As shown in Figure 3(a), densities of 0.20%, 0.39%, 0.78%, 1.56%, and 3.13% correspond to , respectively. SWIFT achieves its highest average accuracy of 88.05% at . Increasing the update density beyond this setting does not further improve accuracy in this experiment. These results support as a practical default for the evaluated settings.
Figure 3. Ablation study of SWIFT. (a) Update-density sensitivity on eight commonsense reasoning datasets with LLaMA3-8B. Densities of 0.20%, 0.39%, 0.78%, 1.56%, and 3.13% correspond to , respectively. (b) Comparison of permutation routing schemes on five image classification datasets with CLIP ViT-B/32. DC, AC, SS, RP, and US denote diagonal cyclic, anti-diagonal cyclic, single-row swap, random permutation, and uniform scattering, respectively.
Permutation type. We further compare routing schemes in Figure 3(b), including diagonal cyclic (DC), anti-diagonal cyclic (AC), single-row swap (SS), random permutation (RP), and uniform scattering (US). US achieves the highest average accuracy of 86.78%, outperforming RP by 0.77 percentage points and DC by 1.71 percentage points. These results support the benefit of the deterministic interleaving construction, which distributes locally mixed outputs across feature groups.
Additional analyses. On LLaMA3-8B, SWIFT improves over DiaBlo by 1.24, 0.93, and 1.05 percentage points for attention-only, feed-forward-only, and combined placement, respectively, with matched budgets within each placement. Across seeds 6, 24, and 42, SWIFT achieves 88.05% mean accuracy with a sample standard deviation of 0.10 percentage points. Appendix C provides the complete placement and seed results, together with the training-loss comparison.
Practical Efficiency
We conduct an additional efficiency comparison on LLaMA3-8B. All compared methods use a matched trainable-parameter budget of 0.88% and are trained on a single NVIDIA Tesla V100 GPU with 32 GB of memory using FP16 precision. The comparison uses the same data, splits, and optimizer settings, with method-specific training durations: SWIFT is trained for two epochs and DiaBlo for three epochs. Results are averaged over seeds 6, 24, and 42. Memory cost denotes the maximum single-GPU training memory under the shared measurement protocol. Training time and FLOPs denote the total wall-clock time and cumulative floating-point operations over each method’s complete training run.
As shown in Figure 4, under the respective training schedules, SWIFT has the shortest total training time and the lowest cumulative FLOP count among the compared implementations, reducing training time by 21.8–60.4% and total training FLOPs by 18.5–33.3% relative to the eight baselines. Its memory cost is 21,040 MB, only 16 MB above the minimum achieved by DiaBlo.
Figure 4. End-to-end efficiency comparison on LLaMA3-8B at a matched 0.88% trainable-parameter budget on one V100-32GB GPU. (a) Total training time. (b) Total training FLOPs. (c) Peak training memory. Training durations follow the method-specific schedules described in the text. Results are averaged over three seeds; lower values are better.
For merged-model inference, the effective weight is formed once as , after which the original dense linear layer is used. In the additional single-GPU LLaMA3-8B commonsense-reasoning inference comparison with batch size 1, SWIFT and the evaluated mergeable baselines achieve the same reported throughput of 6.83 samples/s.
Conclusion
We proposed SWIFT, a sparse fine-tuning framework inspired by small-world connectivity. By combining block-diagonal sparse updates with a fixed deterministic scattering permutation, SWIFT performs local feature mixing followed by cross-group routing under a low parameter budget. We further provided a topology-driven analysis showing that certain random sparse-update models and block-diagonal updates can exhibit information islands, whereas SWIFT guarantees positive block-level expansion under the stated structural and scattering conditions. Extensive experiments on commonsense reasoning, natural language generation, and image classification demonstrate that SWIFT achieves higher average performance than the evaluated low-rank and sparse fine-tuning baselines in each task category. Overall, SWIFT offers both an effective PEFT method and a graph-theoretic perspective for designing sparse update structures with improved cross-group connectivity.
Limitations
SWIFT achieves strong results across language and vision tasks through structured scattering. Its theoretical expansion guarantees require equal-sized, structurally dense square blocks and a sufficient scattering condition. Building on this foundation, exploring more flexible connectivity patterns while preserving parameter efficiency, and extending the theoretical guarantees to more general settings, are promising directions for future work.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 62372132 and in part by the Major Key Project of Pengcheng Laboratory under Grant PCL2025A14. The authors would also like to thank Huawei Ascend Cloud Ecological Development Project for providing high-performance Ascend 910 processors. The authors declare no competing interests.
Datasets and Experimental Settings
Datasets
Commonsense Reasoning Tasks. We evaluate the commonsense reasoning capabilities of SWIFT on the following datasets:
BoolQ Clark et al. [2019]: A question-answering dataset comprising yes/no questions.
PIQA (Physical Interaction Question Answering) Bisk et al. [2020]: A dataset of questions accompanied by two candidate solutions, where identifying the correct answer requires physical commonsense reasoning.
SIQA (Social IQA) Sap et al. [2019]: A question-answering dataset focused on reasoning about motivations, reactions, and social implications in everyday situations.
HellaSwag Zellers et al. [2019]: A commonsense sentence-completion benchmark in which each context is accompanied by multiple candidate endings.
WinoGrande Sakaguchi et al. [2021]: A fill-in-the-blank benchmark with two candidate options for each sentence.
ARC-e/ARC-c (AI2 Reasoning Challenge–Easy/Challenge) Clark et al. [2018]: The Easy and Challenge subsets of ARC, containing multiple-choice questions about school-level science.
OBQA (OpenBookQA) Mihaylov et al. [2018]: A question-answering dataset requiring reasoning that combines scientific facts with additional common knowledge.
Following Hu et al. [2023], we use Commonsense-170K for training. This dataset contains approximately 170,000 examples in a standardized instruction format, covering diverse aspects of commonsense reasoning.
Natural Language Generation Tasks. We evaluate the natural language generation capabilities of SWIFT on the following datasets:
GSM8K Cobbe et al. [2021]: A mathematical reasoning dataset comprising high-quality, linguistically diverse grade-school mathematical word problems written by human problem creators.
MATH Hendrycks et al. [2021]: A mathematical reasoning dataset comprising competition mathematics problems across seven subject categories.
HumanEval Chen et al. [2021]: A code generation benchmark that evaluates the functional correctness of generated Python programs using unit tests.
MBPP Austin et al. [2021]: A benchmark of crowdsourced Python programming problems intended to be approachable for novice programmers, covering basic programming concepts and standard library functionality.
Following Meng et al. [2024], we use the following two datasets for training:
Math: MetaMathQA Yu et al. [2023]. MetaMathQA is a synthetic dataset developed to improve mathematical reasoning in large language models. We use 100K examples from the MetaMathQA training data prepared by PiSSA and evaluate the resulting model on GSM8K and MATH.
Code: Code-Feedback Zheng et al. [2024]. We use 100K examples from the Python instruction-response data prepared by PiSSA from CodeFeedback and evaluate the resulting model on HumanEval and MBPP.
For both training datasets, the released scripts select the first 100K examples from the corresponding training split.
Image Classification Tasks. We evaluate SWIFT on the following image classification datasets:
Cars (Stanford Cars) Krause et al. [2013]: A car classification dataset containing 16,185 images across 196 classes, with separate training and test sets.
DTD (Describable Textures Dataset) Cimpoi et al. [ [53]: A texture classification dataset containing 5,640 images across 47 classes, with 120 images per class.
GTSRB (German Traffic Sign Recognition Benchmark) Stallkamp et al. [ [54]: A traffic sign classification dataset containing over 50,000 images across 43 categories.
RESISC45 Cheng et al. [2017]: A remote sensing scene classification dataset containing 31,500 images across 45 categories, with 700 images per category.
SVHN (Street View House Numbers) Netzer et al. [ [55]: A digit classification dataset derived from Google Street View images, containing 10 classes, with 73,257 training images, 26,032 test images, and an additional set of 531,131 images. Our implementation uses the official training and test splits without the additional set.
Following Fan et al. [ [31], we report classification accuracy on all five datasets.
Experimental Settings
Implementation Details. SWIFT is implemented in PyTorch using Hugging Face Transformers. We use LLaMA3-8B for commonsense reasoning, LLaMA2-7B for natural language generation, and CLIP ViT-B/32 for image classification. The released SWIFT training scripts default to one GPU with FP16 precision for commonsense reasoning, two GPUs with BF16 precision for natural language generation, and four GPUs with BF16 mixed precision for image classification. The distributed generation and image-classification configurations use DeepSpeed ZeRO Stage 2.
The pre-trained backbone weights remain frozen during SWIFT fine-tuning. The trainable block-diagonal updates are initialized to zero, and the scattering permutation is fixed throughout training. For image classification, the classification head is trained jointly with the SWIFT updates.
All downstream experiments, including the main comparisons, ablations, and efficiency evaluations, use random seeds 6, 24, and 42, and the reported results are averaged across the three runs.
Hyperparameters. For SWIFT, the number of blocks is set to 64 in all main experiments; the block-number ablation varies this setting explicitly. We use the same deterministic scattering construction across tasks without task-specific permutation search. SWIFT uses AdamW with zero weight decay and does not apply dropout within its update branch. Detailed task-specific settings are provided below.
For commonsense reasoning, full fine-tuning uses a learning rate of . LoRA, DoRA, PiSSA, and HydraLoRA use a learning rate of with rank 32. SpIEL and SMT follow the hyperparameter settings reported in the SMT paper, and SFT follows the settings reported in the SFT paper. SpaRTA, DiaBlo, MoRe, and SWIFT use a learning rate of .
SWIFT adapts the query, key, and value projections and the feed-forward up and down projections, corresponding to q_proj, k_proj, v_proj, up_proj, and down_proj. Training lasts two epochs with a global batch size of 16, a per-device micro-batch size of 2, and eight gradient accumulation steps in the default single-GPU configuration. The maximum sequence length is 256 tokens, and the learning-rate schedule uses 100 warmup steps. Prompt tokens are excluded from the training loss. We hold out 120 examples from the training data for validation using split seed 42. Validation and checkpoint saving occur every 80 optimizer steps, and the checkpoint with the lowest validation loss is selected.
For natural language generation, full fine-tuning follows the hyperparameter settings used in the PiSSA paper. LoRA, DoRA, and PiSSA use a learning rate of with rank 160, following PiSSA. HydraLoRA uses a learning rate of with rank 32. SpaRTA, DiaBlo, MoRe, and SWIFT use a learning rate of .
SWIFT adapts all attention projections and the feed-forward gate, up, and down projections, corresponding to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. Math and code models are fine-tuned separately for one epoch. The maximum sequence length is 512 tokens. The default configuration uses a per-device micro-batch size of 1 and two gradient accumulation steps on two GPUs, giving a global batch size of 4. We use a cosine learning-rate schedule with a warmup ratio of 0.03 and evaluate the final checkpoint.
For image classification, full fine-tuning, LoRA, DoRA, and PiSSA follow the hyperparameter settings used in the GOAT paper, with a learning rate of and rank 8 for the low-rank methods. SpaRTA, DiaBlo, and MoRe use a learning rate of . SWIFT uses a learning rate of .
SWIFT adapts the linear layers of the vision backbone, including the attention and feed-forward projections. The classification head is trained directly. Training lasts 35, 76, 11, 15, and 4 epochs for Cars, DTD, GTSRB, RESISC45, and SVHN, respectively. The default configuration uses a global batch size of 512, a per-device micro-batch size of 64, and two gradient accumulation steps on four GPUs. We use a cosine learning-rate schedule with a warmup ratio of 0.03.
The vision data loader reserves 10% of each supplied training split, capped at 5,000 examples, using split seed 0, and uses the remaining examples for optimization. Training images undergo random resized cropping to pixels, with a crop scale of , followed by CLIP normalization. Evaluation images are resized and center-cropped to pixels using the same normalization. Evaluation accuracy is recorded at the end of each epoch, and the final adapter and classification head are saved after training. All SWIFT training scripts complete the specified epoch schedule without early stopping.
Evaluation Settings. For commonsense reasoning, the released evaluation script uses a batch size of 1, four-beam decoding, and a maximum of 32 generated tokens. Predictions are extracted using the task-specific answer formats and evaluated by accuracy. For natural language generation, the math and code evaluation scripts use greedy decoding with a batch size of 4 and a maximum of 1,024 generated tokens. Mathematical answers are extracted and compared with the references using the provided evaluation utilities. Code generation is evaluated with EvalPlus using one generated completion per problem.
Proof of Small-World Induced Fine-Tuning Theory
This appendix proves Theorems 3.3, 3.4, and 3.6, together with the MRGC bound in Equation (15) and the coordinate-level expansion bound in Equation (16). Throughout, graphs are induced by the structural support of trainable updates, including trainable entries initialized to zero. Under the row-vector convention, an edge represents a possible direct trainable interaction from input coordinate to output coordinate . We use
The quantities and , defined in Equations (9) and (13), measure directed out-vertex expansion over coordinate subsets and block unions, respectively.
Additional Coverage and Expansion Bounds
The inverse permutation satisfies the same scattering bound, since
Consequently, this condition controls the output routing under the convention in Equation (4).
For a union of blocks, structural density gives and . The scattering condition bounds its overlap with by , yielding (14). The same condition also provides a separate remote-group coverage bound:
Indeed, every source coordinate has output neighbors, with at most in any target block, so it reaches at least blocks, at most one of which is local.
Coordinate-level expansion. Under the assumptions of Theorem 3.6, a separate coordinate-level bound holds:
This follows by considering how many blocks a coordinate subset intersects and using the structurally dense support within each block. The block-level theorem, MRGC bound, and coordinate-level bound are derived separately from the same scattering condition. Detailed proofs are provided in Appendix B.4.
Under exact uniform scattering, , Theorem 3.6 gives . For other compatible dimensions, the realized value in Equation (5) determines whether the sufficient condition holds. Permutations that only exchange entire blocks have ; in particular, swapping disjoint block pairs leaves closed block unions when . The expansion guarantees above apply to equal-sized, structurally dense square blocks, while rectangular projections are characterized through their bipartite group coverage.
Proof of Theorem 3.3: Connectivity Collapse under Independent Random Sparsification
Let be the interaction graph induced by the random structural support, and let denote the out-degree of vertex . Write
The uniform lower-bound assumption
implies that there exist constants and such that
Let
be the number of zero-out-degree vertices. Since outgoing supports are sampled independently across vertices, the events are independent. Therefore, for ,
Thus, .
For , if a vertex has zero out-degree, then the singleton is admissible in Equation (9) and satisfies
Consequently,
It follows that
This proves Theorem 3.3. ■
For the independent Bernoulli model with entry probability , the expected out-degree is and
Hence this model satisfies the theorem’s assumptions. The proof applies more generally whenever outgoing supports are independent across vertices and the zero-out-degree probabilities satisfy the stated uniform lower bound.
Proof of Theorem 3.4: Zero Expansion of Block-Diagonal Updates
Let have block-diagonal structural support under the partition
Thus,
Consider . Since ,
For any source vertex , an edge requires to belong to the same block as . Therefore , giving
Substituting this admissible set into Equation (9) yields
Hence , proving Theorem 3.4. ■
Proof of Theorem 3.6: Expansion of Small-World Induced Fine-Tuning
Let be a partition of into blocks of equal size . The trainable block-diagonal factor has structurally dense blocks:
Let be the permutation matrix associated with under the convention
and write
Output neighborhoods and inverse scattering. For any source coordinate ,
Structural density therefore gives the exact neighborhood identity
In particular, every source coordinate has exactly neighbors in the structural support.
The scattering assumption also holds for . Indeed, for all ,
Block-level expansion. Consider any admissible block union
By Equation (22),
Since is a bijection,
The sets form a partition of , as do the sets . Consequently, the following intersections are pairwise disjoint, and (23) gives
Combining Equations (25) and (26),
Dividing by yields
Taking the minimum over all admissible block unions gives
Under the sufficient condition , this lower bound is strictly positive. This proves Theorem 3.6.
Proof of the MRGC bound. We next prove (15) using the same partition, so that in the MRGC definition. Fix a source coordinate , and let
be the set of all target groups reached by , including its local group if reached. By Equations (22) and (23),
Therefore,
which implies
Excluding the local group removes at most one group, so
Since this holds for every source coordinate,
This proves (15).
Proof of the coordinate-level expansion bound. We finally prove (16). Let be any set satisfying
Define
Every coordinate in a given source block has the same output neighborhood by (22). Since intersects every block indexed by ,
We consider two cases.
Case 1: . Since , the overlap estimate in Equation (26), applied to , gives
Consequently,
The assumption implies
Using , we obtain
Case 2: . Using Equation (31) and the bound , we have
Since ,
The integer satisfies , and hence
Combining the two cases, every admissible coordinate subset satisfies
Taking the minimum over gives
This proves Equation (16). ■
Additional Analyses
Baseline Descriptions
To assess the effectiveness of SWIFT, we perform a comprehensive evaluation against several baseline methods, grouped into three categories:
Full Fine-Tuning: This baseline updates all pre-trained parameters, providing a reference for adaptation without restricting the trainable parameter budget.
Low-Rank Adaptation Family: This group includes LoRA [14] and its variants. We include DoRA [16] for weight-decomposed adaptation, PiSSA [17] for principal-component initialization, GOAT [31] for SVD-structured mixture-of-experts adaptation and optimization alignment, and HydraLoRA [30] for asymmetric adaptation to heterogeneous data. These methods achieve parameter efficiency through low-rank parameterizations and their extensions.
Sparse and Structured Fine-Tuning Approaches: This group includes random, dynamic, and structured tuning methods. We compare SWIFT against SpaRTA [27], which updates randomly selected parameters, and S2FT [22], which selects attention heads and feed-forward channels for structured sparse updates. We also include SMT [33], which selects task-relevant submatrices using saliency signals, and SpIEL [34], which dynamically reallocates sparse trainable parameters. Finally, we compare with MoRe [24], which uses Monarch-style structured parameterizations with fixed permutations, and DiaBlo [25], which finetunes diagonal blocks of selected weight matrices.
Support Simulation and Rectangular Coverage
For rectangular layers, we use separate input and output vertex sets with paired input and output groups. The remote target groups for an input coordinate are the output groups other than its paired group, and the normalization is . The square-graph expansion results in § 3.3 apply to square layers.
We use MRGC to compare representative structural supports under a matched sparse budget, with full fine-tuning as a dense reference. We simulate feature dimensions partitioned into equal-sized groups. The random sparse baseline uses the same expected density as block-diagonal tuning, , and results are averaged over 1,000 sampled masks. Full fine-tuning uses a dense support, block-diagonal tuning retains only within-group interactions, and SWIFT uses the support of BP with the deterministic interleaving permutation.
Full fine-tuning achieves the maximal MRGC of 1.00, while block-diagonal tuning yields 0.00 due to the absence of cross-group interactions. Random sparse tuning improves coverage but remains uneven, achieving a mean MRGC of 0.41. In this setting, SWIFT attains an MRGC of 1.00 at density . Thus, the scattering permutation provides complete remote-group coverage in this configuration without increasing the trainable parameter budget.
Placement and Seed Analyses
Attention and feed-forward placement. We additionally compare SWIFT and DiaBlo when updates are applied to attention projections, feed-forward projections, or both. Within each placement, the two methods use identical target modules and matched trainable-parameter budgets. Results are averaged over the eight commonsense reasoning datasets and seeds 6, 24, and 42. As shown in Figure 5a, SWIFT improves average accuracy by 1.24, 0.93, and 1.05 percentage points, respectively. The improvement across all three placements supports the applicability of permutation-based routing to both module families.
Variation across random seeds. We additionally evaluate SWIFT on LLaMA3-8B using seeds 6, 24, and 42. The average accuracies across the eight commonsense reasoning datasets are 87.94%, 88.10%, and 88.11%, respectively, as shown in Figure 5b. The mean accuracy is 88.05%, with a sample standard deviation of approximately 0.10 percentage points, computed across the three seed-level averages using the denominator with . These statistics describe variation across random seeds in this setting. No additional statistical significance test was performed.

(a) Attention and FFN placement. LLaMA3-8B · matched budget within placement. DiaBlo; SWIFT. Average accuracy (%): Attention-only, DiaBlo 85.58 and SWIFT 86.82, gain +1.24 pp; FFN-only, DiaBlo 86.72 and SWIFT 87.65, gain +0.93 pp; Both, DiaBlo 87.00 and SWIFT 88.05, gain +1.05 pp.

(b) Variation across seeds. LLaMA3-8B · 8 commonsense tasks. Mean: 88.05; SWIFT. Seed 6: 87.94%; seed 24: 88.10%; seed 42: 88.11%. Random seed · sample SD: 0.10 pp.
Figure 5. Placement and seed analyses on LLaMA3-8B. (a) SWIFT and DiaBlo use identical target modules and matched parameter budgets within each placement; scores are averaged over eight commonsense reasoning datasets and three seeds. (b) SWIFT accuracy for seeds 6, 24, and 42, with the horizontal line showing their mean.
Convergence Analysis
As shown in Figure 6, we compare the training loss curves of SWIFT, LoRA, and SpaRTA over two epochs. SWIFT reaches a lower final training loss than both baselines and maintains a low loss during the later stage of training. These curves support favorable optimization behavior under the evaluated training schedule. Wall-clock training efficiency is assessed separately in § 4.5.

Figure 6. Training loss curves for LoRA, SpaRTA, and SWIFT when fine-tuning LLaMA3-8B on Commonsense-170K for commonsense reasoning.
Broader Impacts and Asset Information
Broader Impacts
SWIFT reduces the number of trainable parameters required for adapting pre-trained models and demonstrates lower training costs in the evaluated efficiency comparison. These benefits may make task-specific adaptation more accessible to researchers with limited computational resources. Adapted models may inherit biases, factual inaccuracies, or other undesirable behaviors from their pre-trained backbones and training data. Parameter-efficient adaptation alone does not address these issues. Downstream deployment therefore requires application-specific evaluation and safeguards appropriate to the intended use.
Asset Attribution and Licenses
The original model and dataset publications are cited in the main text and Appendix § 8.§ 1. LLaMA3-8B is distributed under the Meta Llama 3 Community License Agreement, and LLaMA2-7B under the Llama 2 Community License Agreement. The official CLIP implementation uses the MIT License, and PEFT uses the Apache License 2.0.
The SWIFT repository retains upstream license files and third-party notices, including the MIT license for the GOAT-derived image-classification code. Applicable subdirectory and file-level notices remain in effect. Pre-trained model weights and datasets are obtained separately from their original providers under the corresponding access conditions and terms of use; they are not redistributed with the SWIFT code.
Code Availability
The SWIFT implementation is available at https://github.com/xianchaoguan/SWIFT, including training and evaluation scripts, environment specifications, and data-preparation instructions for commonsense reasoning, natural language generation, and image classification. The release covers the SWIFT pipelines for these three task groups; it does not include dedicated reproduction scripts for all baseline comparisons, ablations, and efficiency measurements. Pre-trained weights and datasets must be obtained separately from their original sources under the applicable access conditions.
References
- [1]Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. Natural language understanding and inference with mlm in visual question answering: A survey. ACM Computing Surveys, 57(8):1–36, 2025.arxiv.org/abs/2411.17558
- [2]Agnes Axelsson and Gabriel Skantze. Using large language models for zero-shot natural language generation from knowledge graphs. arXiv preprint arXiv:2307.07312, 2023.arxiv.org/abs/2307.07312
- [3]Weikang Meng, Yadan Luo, Liangyu Huo, Yingjian Li, Yaowei Wang, Zheng Zhang, and Heng Tao Shen. Polaformer++: Polarity-aware linear attention with channel-wise spikiness. IEEE Trans. Pattern Anal. Mach. Intell., 48(11):15412–15426, 2026. doi: 10.1109/TPAMI.2026.3723414. URL https://doi.org/10.1109/TPAMI.2026.3723414.
- [4]Weikang Meng, Yadan Luo, Liangyu Huo, Yaowei Wang, Xin Li, and Zheng Zhang. Nalaformer: Norm-aware linear attention for transformer models. CoRR, abs/2506.21137, 2025. doi: 10.48550/ARXIV.2506.21137. URL https://doi.org/10.48550/arXiv.2506.21137.
- [5]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024a.arxiv.org/abs/2412.15115
- [6]Cong Jiang, Xiaofeng Zhang, Fangzhi Zhu, XiaoWei Chen, Junxiong Zhu, and Zheng Zhang. Rethinking LLM reasoning: From explicit trajectories to latent representations. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=CbK7lYbmv8.
- [7]Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023a.arxiv.org/abs/2308.10792
- [8]Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Khan, and Arsalan Shahid. The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities. arXiv preprint arXiv:2408.13296, 2024.arxiv.org/abs/2408.13296
- [9]Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024a.arxiv.org/abs/2403.03507
- [10]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
- [11]Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. arXiv preprint arXiv:2402.09997, 2024b.arxiv.org/abs/2402.09997
- [12]Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiven Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning. Advances in Neural Information Processing Systems, 37:71768–71791, 2024b.arxiv.org/abs/2406.05223
- [13]Dan Zhang, Tao Feng, Lilong Xue, Yuandong Wang, Yuxiao Dong, and Jie Tang. Parameter-efficient fine-tuning for foundation models. arXiv preprint arXiv:2501.13787, 2025.arxiv.org/abs/2501.13787
- [14]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
- [15]Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023b.arxiv.org/abs/2303.10512
- [16]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.09353
- [17]Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37: 121038–121072, 2024.arxiv.org/abs/2404.02948
- [18]Cong Jiang, Fangzhi Zhu, Xiaowei Chen, Junxiong Zhu, Bo Zheng, Yifeng Wang, and Zheng Zhang. Dmlora: Dynamic multi-subspace low-rank adaptation. In Companion Proceedings of the ACM on Web Conference 2025, pages 1038–1042, 2025.DOI
- [19]Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024.arxiv.org/abs/2402.12354
- [20]Zhiwei He, Zhaopeng Tu, Xing Wang, Xingyu Chen, Zhijie Wang, Jiahao Xu, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, and Rui Wang. Rasa: Rank-sharing low-rank adaptation. arXiv preprint arXiv:2503.12576, 2025a.arxiv.org/abs/2503.12576
- [21]Weixi Song, Zuchao Li, Lefei Zhang, Hai Zhao, and Bo Du. Sparse is enough in fine-tuning pre-trained large language models. arXiv preprint arXiv:2312.11875, 2023.arxiv.org/abs/2312.11875
- [22]Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen. S²ft: Efficient, scalable and generalizable llm fine-tuning by structured sparsity. Advances in Neural Information Processing Systems, 37:59912–59947, 2024c.arxiv.org/abs/2412.06289
- [23]Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. Parameter-efficient fine-tuning in large language models: a survey of methodologies. Artificial Intelligence Review, 58(8):227, 2025.DOI
- [24]Wenxuan Tan, Nicholas Roberts, Tzu-Heng Huang, Jitian Zhao, John Cooper, Samuel Guo, Chengyu Duan, and Frederic Sala. More fine-tuning with 10x fewer parameters. arXiv preprint arXiv:2408.17383, 2024.arxiv.org/abs/2408.17383
- [25]Selcuk Gurses, Aozhong Zhang, Yanxia Deng, Xun Dong, Xin Li, Naigang Wang, Penghang Yin, and Zi Yang. Diablo: Diagonal blocks are sufficient for finetuning. arXiv preprint arXiv:2506.03230, 2025.arxiv.org/abs/2506.03230
- [26]Chaitanya K Joshi. Transformers are graph neural networks. arXiv preprint arXiv:2506.22084, 2025.arxiv.org/abs/2506.22084
- [27]Jesus Rios Aliaga, Pierre Dognin, Ronny Luss, and Karthikeyan Natesan Ramamurthy. Sparsity may be all you need: Sparse random parameter adaptation. In Conference on Empirical Methods in Natural Language Processing, 2025.
- [29]João Pedro C Morais and Ruben Interian. A simple and flexible algorithm to generate real-world networks. In International Conference on Complex Networks, pages 1–12. Springer, 2025.arxiv.org/abs/2502.18579
- [30]Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37: 9565–9584, 2024.arxiv.org/abs/2404.19245
- [31]Chenghao Fan, Zhenyi Lu, Sichen Liu, Chengfeng Gu, Xiaoye Qu, Wei Wei, and Yu Cheng. Make lora great again: Boosting lora with adaptive singular values and mixture-of-experts optimization alignment. arXiv preprint arXiv:2502.16894, 2025.arxiv.org/abs/2502.16894
- [32]Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513, 2023. URL https://arxiv.org/abs/2310.17513.
- [33]Haoze He, Juncheng B Li, Xuan Jiang, and Heather Miller. Smt: Fine-tuning large language models with sparse matrices. In The Thirteenth International Conference on Learning Representations, 2025b.
- [34]Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401.16405, 2024.arxiv.org/abs/2401.16405
- [35]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
- [36]Zhiqiang Hu, Lei Wang, Yihui Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933, 2023.arxiv.org/abs/2304.01933
- [37]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.arxiv.org/abs/2307.09288
- [38]Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023.arxiv.org/abs/2309.12284
- [39]Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658, 2024.arxiv.org/abs/2402.14658
- [40]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021.arxiv.org/abs/2103.00020
- [53]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014.arxiv.org/abs/1311.3618
- [54]Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks, pages 1453–1460. IEEE, 2011.DOI
- [55]Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. URL https://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf.



