betaFeedback

WorksComputer Science

FedLoVA: Value-Only Aggregation for Federated LoRA Fine-Tuning of Large Language Models

Abstract

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for fine-tuning of large language models (LLMs), especially in federated learning, due to its strong performance and communication efficiency. In practice, LoRA is applied to the query and value projections of transformer layers without considering the distinct roles of these components. In this work, we analyze the impact of these projections in federated fine-tuning of LLMs using LoRA and observe that the value projection converges significantly faster than the query projection. Motivated by this finding, we propose FedLoVA, a federated fine-tuning framework that trains LoRA adapters on both query and value projections locally while sharing only the value projection with the server. This approach reduces communication overhead while improving convergence efficiency. Experiments on natural language understanding, multilingual sentiment analysis, and generation tasks show that FedLoVA reduces the communication cost by up to 7.8×7.8\times compared to existing federated LLM fine-tuning baselines while achieving comparable or better performance. The code is available at: https://github.com/EnsiehKhazaei/FedLoVA.

Introduction

Large Language Models (LLMs) have emerged in recent years and demonstrated remarkable performance in various applications, including conversational systems, finance, and healthcare [38, 9, 37]. To achieve strong performance in these applications, LLMs typically require fine-tuning on task-specific downstream datasets. However, due to their massive number of parameters, full fine-tuning of LLMs requires substantial computational resources and memory capacity, limiting their practical deployment in real-world scenarios with scarce resources.

To address the high computational and memory costs associated with full fine-tuning of LLMs, Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed as promising solutions. Instead of updating all model parameters during fine-tuning, PEFT methods train only a small set of parameters. Some PEFT approaches include Adapter Tuning [20], Prefix-Tuning [33], and Low-Rank Adaptation (LoRA) [34, 29]. Among these methods, LoRA has gained significant attention due to its simplicity and effectiveness. LoRA decomposes weight updates into low-rank matrices AA and BB, and learns these matrices instead of updating the full parameters. This technique achieves competitive performance compared to full model fine-tuning while reducing the number of trainable parameters.

Although LoRA reduces the number of parameters involved in fine-tuning, applying it in resource-constrained environments such as Federated Learning (FL) frameworks remains challenging. In such settings, limited computational resources, communication overhead, client heterogeneity, and privacy concerns must be considered. To address these challenges, several studies have explored federated fine-tuning of LLMs using LoRA [36, 2, 1]. Among these, FFA-LoRA [28] freezes the AA matrix during training and aggregates only the BB matrix across clients. However, freezing AA can limit the performance and adaptability of the FL framework. Another approach, FedSA-LoRA [10], trains both AA and BB locally but shares only AA with the server, based on the intuition that AA captures global knowledge while BB captures personalized knowledge.

Despite these prior studies investigating the roles of LoRA matrices AA and BB in FL frameworks, it remains underexplored how different attention components, specifically the query (QQ) and value (VV) projections, affect global and local performance in federated settings. Understanding the roles of these attention modules is crucial in federated fine-tuning of LLMs, where communication efficiency, convergence behavior, and privacy preservation are key constraints. Identifying which attention modules are most beneficial to share with the global server versus to keep local can directly lead to more efficient and effective federated fine-tuning approaches.

In this work, we provide a comprehensive study of attention submodules in federated LLM fine-tuning, quantifying their impact on global generalization and local adaptation. In particular, we analyze how sharing or localizing each component affects performance, convergence behavior, and discuss potential implications for privacy in FL frameworks. Through this analysis, we identify the value (VV) projection in the attention mechanism as the most critical module for aggregation among clients. Based on this insight, we propose FedLoVA (Federated LoRA with Value Aggregation), a communication-efficient framework for federated fine-tuning of LLMs using LoRA. FedLoVA applies LoRA adapters to both the query (QQ) and value (VV) projections within each transformer layer, trains both modules locally, and shares only the value-related adapter parameters (AvA_v and BvB_v matrices) with the server for aggregation, as shown in Figure 1. This sharing strategy reduces communication overhead and limits the information shared with the server, although we do not claim formal privacy guarantees.

Illustration of communication and update strategies in FedLoRA, FFA-LoRA, FedSA-LoRA, and FedLoVA

Figure 1. Illustration of communication and update strategies in a) FedLoRA, b) FFA-LoRA, c) FedSA-LoRA, and d) the proposed FedLoVA. FedLoRA updates and shares both AA and BB for query and value projections. FFA-LoRA freezes AA and only updates and shares BB. FedSA updates both AA and BB locally but shares only AA, while FedLoVA updates both AA and BB and shares value projection modules AvA_v and BvB_v.

Further, data heterogeneity is a well-known critical challenge in FL. In federated fine-tuning of LLMs, this challenge arises from class-label heterogeneity and linguistic variation, since clients in realistic scenarios may have text in different languages. This linguistic heterogeneity is a source of non-IID data specific to the text setting. Prior work has not examined how these modules behave under multilingual heterogeneity, which is particularly relevant in federated settings for LLMs. In this work, we study how attention modules perform when clients operate in multilingual environments, providing a broader understanding of LoRA adaptation under both class-label and linguistic heterogeneity to test whether the value-aggregation finding holds across different and realistic sources of non-IID data.

We summarize our main contributions as follows:

  • We investigate the relationship between attention projections (QQ and VV) under federated fine-tuning of LLMs using LoRA. We analyze how the FL framework behaves when different modules are shared with the server compared to fully shared settings.

  • We introduce FedLoVA, a novel federated fine-tuning framework in which only the value projection modules are aggregated by the server. We show that FedLoVA improves convergence behavior and performance compared to existing federated LLM fine-tuning baselines.

  • We conduct comprehensive experiments on five text classification datasets from the GLUE benchmark, a multilingual dataset, and a generative task. Our results show that FedLoVA outperforms prior approaches under both class-label heterogeneity and multilingual heterogeneity, confirming its robustness, communication efficiency, and adaptability.

Related Work

PEFT in Federated Learning (FL)

FL is a distributed machine learning paradigm in which multiple clients collaboratively train a global model without sharing their local data with a central server [13, 14, 15, 18]. With the rapid growth of LLMs, FL has emerged as an appealing framework for fine-tuning these models across distributed data sources. Fine-tuning LLMs typically requires large amounts of data; however, such data are often generated across many sources, and aggregating them on a central server is challenging due to privacy concerns and communication costs. Full fine-tuning of LLMs in federated settings introduces additional challenges, including substantial computational demands on clients and communication overhead.

To address these challenges, PEFT methods have been introduced to adapt large pretrained models by updating only a small subset of parameters or by introducing lightweight trainable components, while keeping the majority of the model frozen. PEFT methods significantly reduce local computation costs and the number of parameters exchanged between clients and the server. PEFT methods can be categorized into different approaches. Adapter-based methods insert small bottleneck layers into each transformer block and train only these modules [11]. In [39], an adapter-based federated multimodal learning framework is proposed that freezes large pretrained backbones and trains lightweight task-and modality-specific adapters via split FL. Prompt-tuning approaches are another line of work that optimize a limited number of continuous prompts with a frozen backbone model [8, 6]. FedBPT [27] is an FL framework that learns shared prompts via gradient-free optimization, which significantly reduces computation costs, and shares only prompts with the server instead of the model parameters, thereby reducing communication overhead. LoRA is another category that has attracted special attention due to its simplicity and competitive performance, which is also the focus of this work. LoRA-based methods reduce the number of communicated parameters through low-rank matrix decompositions [12]. Some FL works [3, 28, 35] employ LoRA, which we discuss next.

LoRA in Federated Learning

LoRA is a PEFT technique that adapts pretrained models by injecting low-rank trainable matrices AA and BB into existing weight matrices, while keeping the original model parameters frozen [12]. LoRA significantly reduces the number of trainable parameters, which is beneficial in bandwidth-limited and resource-constrained federated settings. FDLoRA [24] is an approach that uses dual LoRA modules and a gradient-free adaptive fusion strategy to obtain both global and personalized knowledge. Although this method is beneficial in heterogeneous scenarios, it requires higher computational resources, since two LoRA modules must be trained.

LoRA-FAIR [2] addresses the aggregation inconsistency inherent in standard federated LoRA, where averaging LoRA matrices diverges from the ideal global update due to the nonlinearity of their product. LoRA-FAIR introduces a server-side correction term to the averaged LoRA matrices, improving aggregation accuracy. FFA-LoRA [28] is proposed to address the instability of standard LoRA under privacy-preserving federated learning by fixing AA and training only BB. However, fixing AA can limit the ability to reach the optimal solution. FedSA-LoRA [10] further investigates the roles of AA and BB and finds that AA is responsible for capturing global knowledge, while BB captures personalized knowledge. In FedSA-LoRA, both AA and BB are trained locally, and only the AA matrices containing global knowledge are aggregated at the server. Unlike previous works, which investigated the roles of AA and BB to some extent, we adopt a different perspective and aim to study the role of query and value projections in LoRA adapters within the FL framework.

Problem Statement

LoRA in Transformer Attention Mechanism

LLMs require extensive computational resources for training. LoRA helps reduce the number of trainable parameters during fine-tuning. In transformer-based LLMs, LoRA is typically applied to the query (Q) and value (V) projection matrices within the attention mechanism [12]. Given an input XX, the attention output is computed as:

Attention⁡(Q,K,V)=softmax⁡(QKTdk)V(1)\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\mathsf{T}}}{\sqrt{d_k}}\right)V \tag*{(1)}

where the projections are defined as:

Q=WqX,Wq∈Rdq×din,X∈Rdin,Q∈Rdq(2)Q=W_qX,\quad W_q\in\mathbb{R}^{d_q\times d_{\mathrm{in}}},\quad X\in\mathbb{R}^{d_{\mathrm{in}}},\quad Q\in\mathbb{R}^{d_q} \tag*{(2)}
K=WkX,Wk∈Rdk×din,K∈Rdk(3)K=W_kX,\quad W_k\in\mathbb{R}^{d_k\times d_{\mathrm{in}}},\quad K\in\mathbb{R}^{d_k} \tag*{(3)}
V=WvX,Wv∈Rdv×din,V∈Rdv(4)V=W_vX,\quad W_v\in\mathbb{R}^{d_v\times d_{\mathrm{in}}},\quad V\in\mathbb{R}^{d_v} \tag*{(4)}

Here, dind_{\mathrm{in}} denotes the input feature dimension, dqd_q, dkd_k, and dvd_v are the dimensions of the query, key, and value vectors, respectively. The dot product QKTQK^{\mathsf{T}} produces the attention scores normalized by dk\sqrt{d_k}. WqW_q, WkW_k, WvW_v are trainable weight matrices. In LoRA, WqW_q and WvW_v are expressed as the sum of the pretrained weights and a trainable low-rank update:

Wq=Wq,0+BqAq,Aq∈Rr×din,Bq∈Rdq×r(5)W_q=W_{q,0}+B_qA_q,\quad A_q\in\mathbb{R}^{r\times d_{\mathrm{in}}},\quad B_q\in\mathbb{R}^{d_q\times r} \tag*{(5)}
Wv=Wv,0+BvAv,Av∈Rr×din,Bv∈Rdv×r(6)W_v=W_{v,0}+B_vA_v,\quad A_v\in\mathbb{R}^{r\times d_{\mathrm{in}}},\quad B_v\in\mathbb{R}^{d_v\times r} \tag*{(6)}

where Wq,0W_{q,0} and Wv,0W_{v,0} are the pretrained frozen weights, and r≪dq,dvr\ll d_q,d_v is the LoRA rank. The matrices BqB_q and BvB_v are zero-initialized, while AqA_q and AvA_v are randomly initialized. All LoRA parameters are trained during the fine-tuning procedure to update WqW_q and WvW_v.

Empirical Motivation for Query and Value Projections

Following the original LoRA paper [12], it has been shown that training both QQ and VV matrices simultaneously provides the best performance under a fixed parameter budget. Hu et al. [12] also studied the cases where LoRA is applied to only QQ or only VV in centralized settings, and found that assigning all trainable parameters to either QQ or VV leads to significantly lower performance, whereas adapting both yields the best results. We also provide a comprehensive analysis of LoRA applied to different attention modules and their combinations in Appendix § 7.§ 2 to justify the effectiveness of the Q+VQ+V design choice. However, [12] did not investigate how these matrices affect the convergence behavior of the model. To this end, we first analyze the convergence behavior of QQ and VV independently. Specifically, we apply LoRA with rank r=8r=8 once on the QQ and once on the VV projection and evaluate performance on two datasets from the GLUE benchmark, namely SST2 and MNLI-mm, using the RoBERTa-large model with 355M parameters. As shown in Figure 2, LoRA applied to the VV projection exhibits much faster convergence than when applied to the QQ projection. To quantify this, we define the target accuracy as 95% of the final accuracy achieved by the QQ projection (89.76% for SST2 and 80.65% for MNLI-mm), and measure the convergence speed multiplier as the ratio of communication rounds required by QQ to reach the target accuracy relative to that required by the VV projection. The value projection reaches in 3.3×3.3 \times fewer rounds on SST2 and 2.4×2.4 \times fewer rounds on MNLI-mm to the target accuracy. We argue that this difference stems from a structural asymmetry in how QQ and VV interact with the optimization landscape. When optimizing VV alone, the attention weights are fixed for QQ and KK, making the model output approximately linear in VV. The loss can thus be viewed as the composition of a linear map and the downstream objective, which is expected to yield a better-conditioned optimization landscape with more stable, well-aligned gradients, and consequently faster convergence. When optimizing QQ alone, the situation is different, QQ influences the model output indirectly through the softmax operation, introducing strong nonlinearity. The resulting gradients may be poorly conditioned, particularly when attention logits fall into saturated regimes, leading to greater sensitivity and likely slowing convergence. To verify that this observation is not dependent on model scale or task type, we conduct the same experiments using a smaller model (BERT with 110M parameters) on the GLUE benchmark and a larger model (Llama2 with 7B parameters) on a generative task, with results reported in Appendix § 7.§ 3. For generative task, we evaluate on the GSM8K dataset, a mathematical reasoning benchmark, to assess whether this behavior generalizes across task types. This observation motivates our hypothesis that, although training both QQ and VV can achieve the best performance, sharing only VV with the global server during aggregation may yield faster convergence and better communication efficiency, since QQ converges much more slowly and may contribute less effectively in federated aggregation.

Convergence behavior of LoRA applied separately to Q and V matrices using RoBERTa-large on SST2 and MNLI-mm datasets

Figure 2. Convergence behavior of LoRA applied separately to QQ and VV matrices using RoBERTa-large on SST2 and MNLI-mm datasets. Convergence speed multipliers are computed based on reaching 95% of the final accuracy of QQ, which is 89.76% for SST2 and 80.65% for MNLI-mm.

Theoretical Motivation for Sharing the Value Projection

The convergence gap observed in § 3.2 is consistent with recent theoretical results on the curvature of self-attention. Ormaniec et al. [25] derive the exact loss Hessian of a self-attention layer in closed form and decompose it, via the Gauss–Newton split, into an outer-product term HoH_o and a functional term HfH_f, where the latter can be indefinite. Building on their analysis, we identify two asymmetries between the query and value projections. Note that we consider the curvature with respect to each projection matrix separately, i.e., while all other parameters are held fixed, and with respect to the full matrix rather than its low-rank LoRA factors.

Curvature asymmetry. Because the output of self-attention is linear in WvW_v, the functional term vanishes on the value block, Hf(Wv,Wv)=0H_f(W_v,W_v)=0 [25 Theorems 3.1–3.2], and hence H(Wv,Wv)=Ho(Wv,Wv)H(W_v,W_v)=H_o(W_v,W_v). Since HoH_o is a Gram matrix of the network Jacobian, each block is positive semi-definite for a convex loss (e.g., cross-entropy). In contrast, both Ho(Wq,Wq)H_o(W_q,W_q) and Hf(Wq,Wq)H_f(W_q,W_q) are generically nonzero and depend on the first- and second-order derivatives of the softmax, so the query block is generically indefinite. Ormaniec et al. [25] further show that removing the softmax nonlinearity collapses HfH_f to a block-hollow (all-zero-diagonal) structure and the Hessian blocks become positive semi-definite, whereas the softmax increases its indefiniteness. This asymmetry offers a mechanistic explanation for the slower convergence of QQ, as gradient-based methods are known to converge substantially more slowly near saddle points and plateaus associated with indefinite curvature [7]. Such curvature-induced difficulties cannot arise within a positive semi-definite Hessian block such as H(Wv,Wv)H(W_v,W_v).

Data-dependence asymmetry. The same analysis shows that the query Jacobian scales cubically with the input norm, ∥∂A/∂Wq∥F∈O(∥X∥F3)\lVert\partial A/\partial W_q\rVert_F \in O(\lVert X\rVert_F^3), whereas the value Jacobian scales only linearly, ∥∂F/∂Wv∥F∈O(∥X∥F)\lVert\partial F/\partial W_v\rVert_F \in O(\lVert X\rVert_F) [25]. Consequently, the outer-product Hessian blocks, which are quadratic in these Jacobians, satisfy Ho(Wq,Wq)∈O(∥X∥F6)H_o(W_q,W_q) \in O(\lVert X\rVert_F^6) and Ho(Wv,Wv)∈O(∥X∥F2)H_o(W_v,W_v) \in O(\lVert X\rVert_F^2). In FL, where local data distributions differ across clients, a parameter for which the local curvature depends on XX only quadratically should exhibit less cross-client variation in its local optimization landscape than one with sixth-order dependence. This gives a direct motivation for aggregating VV rather than QQ in federated settings.

Implications for federated aggregation. Taken together, these results indicate that VV is both better conditioned and less sensitive to client-specific data than QQ. For each client kk, holding its locally trained QkQ_k fixed, the curvature of the local objective along VV is positive semi-definite, and its dependence on the client’s data is mild. This makes VV a natural candidate for aggregation, while QQ, characterized by a more indefinite and strongly data-dependent optimization landscape, is better

Algorithm 1 FedLoVA: Federated LoRA with Value Aggregation

References

  1. [1]Sara Babakniya, Ahmed Elkordy, Yahya Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. SLoRA: Federated parameter efficient fine-tuning of language models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023.arxiv.org/abs/2308.06522
  2. [2]Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. LoRA-FAIR: Federated lora fine-tuning with aggregation and initialization refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3737–3746, 2025.arxiv.org/abs/2411.14961
  3. [3]Yuji Byun and Jaeho Lee. Towards federated low-rank adaptation of language models with rank heterogeneity. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 356–362, 2025.
  4. [4]Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. arXiv preprint arXiv:2401.06432, 2024.arxiv.org/abs/2401.06432
  5. [5]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  6. [6]Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. Harmonizing generalization and personalization in federated prompt learning. arXiv preprint arXiv:2405.09771, 2024.arxiv.org/abs/2405.09771
  7. [7]Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems, 27, 2014.arxiv.org/abs/1406.2572
  8. [8]Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompt-tuning in bridging generalized and personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6087–6097, 2024.arxiv.org/abs/2310.18285
  9. [9]Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward. Healai: A healthcare llm for effective medical documentation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 1167–1168, 2024.DOI
  10. [10]Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Ou. Selective aggregation for low-rank adaptation in federated learning. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.01463
  11. [11]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attarian, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019.arxiv.org/abs/1902.00751
  12. [12]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  13. [13]Ensieh Khazaei, Alireza Esmaeilzehi, Bilal Taha, and Dimitrios Hatzinakos. CDFL: efficient federated human activity recognition using contrastive learning and deep clustering. IEEE Sensors Journal, 24(22):38196–38208, 2024.arxiv.org/abs/2407.12287
  14. [14]Ensieh Khazaei and Dimitrios Hatzinakos. Forget less, learn more: Contrastive-based federated class incremental learning with a low-dimensional projection layer. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1782–1791. IEEE, 2025.DOI
  15. [15]Ensieh Khazaei, Bilal Taha, Alireza Esmaeilzehi, and Dimitrios Hatzinakos. OSR: Toward developing efficient federated learning-based human activity recognition using optimal server representations. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025.DOI
  16. [16]Jisoo Kim, Sungmin Kang, and Sunwoo Lee. Layer-wise update aggregation with recycling for communication-efficient federated learning. arXiv preprint arXiv:2503.11146, 2025.arxiv.org/abs/2503.11146
  17. [17]Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5260–5271, 2024.arxiv.org/abs/2309.00363
  18. [18]Seungjun Lee, Ensieh Khazaei, Dimitrios Hatzinakos, Baturalp Buyukates, and Sunwoo Lee. Latent information sharing for accelerating federated learning. arXiv preprint arXiv:2610.01126, 2026.arxiv.org/abs/2610.01126
  19. [19]Huaicheng Li, Junhui Zhao, Haoyu Quan, and Xiaoming Wang. Fed-DLoRA: Efficient wireless federated learning with dynamic low-rank adaptation. IEEE Transactions on Vehicular Technology, 2026.
  20. [20]Minglei Li, Peng Ye, Yongqi Huang, Lin Zhang, Tao Chen, Tong He, Jiayuan Fan, and Wanli Ouyang. Adapter-X: A novel general parameter-efficient fine-tuning framework for vision. arXiv preprint arXiv:2406.03051, 2024.arxiv.org/abs/2406.03051
  21. [21]Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.arxiv.org/abs/1907.02189
  22. [22]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.arxiv.org/abs/1907.11692
  23. [23]Guodong Long, Tao Shen, Jing Jiang, Michael Blumenstein, et al. Dual-personalizing adapter for federated foundation models. Advances in Neural Information Processing Systems, 37:39409–39433, 2024.arxiv.org/abs/2403.19211
  24. [24]Yao Lu, Jiaxing Qi, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian. Fdlora: Personalized federated learning of large language model via dual lora tuning. arXiv preprint arXiv:2406.07925, 2024.arxiv.org/abs/2406.07925
  25. [25]Weronika Ormaniec, Felix Dangel, and Sidak Pal Singh. What does it mean to be a transformer? insights from a theoretical hessian analysis. In International Conference on Learning Representations, volume 2025, pages 17046–17081, 2025.arxiv.org/abs/2410.10986
  26. [26]Raghav Singhal, Kaustubh Ponk she, and Praneeth Vepakomma. FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1316–1336, 2025.DOI
  27. [27]Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yiran Chen, and Holger R Roth. Fedbpt: Efficient federated black-box prompt tuning for large language models. arXiv preprint arXiv:2310.01467, 2023.arxiv.org/abs/2310.01467
  28. [28]Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Representation Learning, volume 2024, pages 17978–17994, 2024.arxiv.org/abs/2403.12313
  29. [29]Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37:9565–9584, 2024.arxiv.org/abs/2404.19245
  30. [30]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971
  31. [31]Linh Tran, Ana Milanova, and Stacy Patterson. Improving parameter-efficient federated learning with differentially private refactorization. arXiv preprint arXiv:2605.08443, 2026.arxiv.org/abs/2605.08443
  32. [32]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.arxiv.org/abs/1804.07461
  33. [33]Haonan Wang, Brian K Chen, Li Siquan, Liang Xinhe, Tianyang Hu, Hwee Kuan Lee, and Kenji Kawaguchi. Prefix-Tuning+: Modernizing prefix-tuning by decoupling the prefix from attention. In Second Workshop on Test-Time Adaptation: Putting Updates to the Test! at ICML 2025, 2025.arxiv.org/abs/2506.13674
  34. [34]Runqian Wang, Soumya Ghosh, David Cox, Diego Antognini, Aude Oliva, Rogerio Feris, and Leonid Karlinsky. Trans-LoRA: towards data-free transferable parameter efficient finetuning. Advances in Neural Information Processing Systems, 37:61217–61237, 2024.arxiv.org/abs/2405.17258
  35. [35]Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. Advances in Neural Information Processing Systems, 37:22513–22533, 2024.arxiv.org/abs/2409.05976
  36. [36]Panlong Wu, Kangshuo Li, Ting Wang, Yanjie Dong, Victor CM Leung, and Fangxin Wang. FedFMSL: Federated learning of foundation models with sparsely activated lora. IEEE Transactions on Mobile Computing, 23(12):15167–15181, 2024.DOI
  37. [37]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling, 2024.arxiv.org/abs/2308.08155
  38. [38]Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yupeng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, et al. Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making. Advances in Neural Information Processing Systems, 37:137010–137045, 2024.arxiv.org/abs/2407.06567
  39. [39]Zixin Zhang, Fan Qi, and Changsheng Xu. Enhancing storage and computational efficiency in federated multimodal learning for large-scale models. In Forty-first International Conference on Machine Learning, 2024.

Paper details

vlnc:W_dcaec9jj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents