betaFeedback

WorksComputer Science

Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks

Abstract

Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates ΔWiΔW_i from client-specific LoRA factors (Ai,Bi)(A_i,B_i) before aggregating them. Although these matrix updates lie in the same weight space Rm×n\mathbb{R}^{m\times n} and can be averaged directly, the LoRA factors that generate them may expressed client-specific local basis. In the typical compose–aggregate–factorize–dispatch loop, this basis inconsistency can make the rank-rir_i reference prefix dispatched to lower-rank clients unstable after server-side refactorization. In this work, we propose FedAbA, a basis-consistent federated LoRA aggregation framework for heterogeneous ranks. FedAbA first extracts each client's local basis, aligns it to the corresponding prefix of the global reference basis, and constructs a basis-consistent reconstruction of each local matrix update before aggregation. It then aggregates these reconstructions with weights that combine client data size and consistency score, followed by exact coefficient-space SVD refactorization and gauge fixing before rank-prefix dispatch. Our theoretical analysis clarifies the role of basis consistency and supports the proposed consistency-aware aggregation mechanism. Extensive experiments show that FedAbA outperforms representative federated LoRA baselines, with the largest gains under the strongest rank heterogeneity.

Paper details

vlnc:W_cefaamwb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
arXiv nonexclusive license
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…