betaFeedback

WorksComputer Science

SPICE: Sparse Parameter Gossip for LLM Pretraining

Abstract

Pretraining language models across geographically distributed resources requires communication-efficient methods that operate under limited and variable internet bandwidth. Decentralized training replaces global collectives with peer exchanges, but full-model messages make larger peer neighborhoods costly. We introduce SPICE, which combines sparse parameter gossip with error feedback through corrected parameter values that remain directly interpretable as peer neighborhoods change. These sparse exchanges make broader neighborhoods affordable, achieving better training quality with less communication than the evaluated dense decentralized baselines. As the replica count grows, SPICEincreases the number of peers contacted while keeping individual messages sparse, limiting the quality degradation observed with dense gossip. Under bandwidth constraints, SPICE achieves higher training throughput than DDP and dense gossip, while sustaining throughput during transient link slowdowns.

Introduction

Training large language models at scale relies on distributing computation across many model replicas, each potentially spanning several GPUs. In standard synchronous data parallel training, these replicas average their gradients before each optimizer step (Li et al., 2020), typically through an All-Reduce collective [34]. Each step therefore requires communication among all replicas and waits for the slowest participant, making training efficiency dependent on the connectivity between them.

Even within tightly connected GPU clusters, communication bandwidth can limit training efficiency, motivating low-precision transfers to reduce communication volume [7]. These constraints become more severe when training extends over geographically distributed data centers, where bandwidth can be limited and vary across participants. Recent large language model pretraining runs over the internet [14, 25] demonstrate the opportunity to pool resources across sites, but exploiting these resources efficiently requires limiting the cost of communication [33, 21].

Decentralized (gossip) training replaces the typical global All-Reduce operation with exchanges among selected peers (Koloskova et al., 2020). This has several potential advantages as it can avoid communication bottlenecks, support heterogeneous bandwidth, and maintain robustness to failure of individual links. However, combining gossip methods with adaptive optimizers has historically been difficult, limiting their applicability to demanding settings such as pretraining foundational models. Recent works by Wang et al. [47] and Cagnasso et al. [6] have shown that decentralized methods can indeed achieve strong performance, narrowing the training quality gap with synchronous baselines in LLM pretraining. In decentralized training, communication topology also affects convergence through its spectral properties [32], while GASLoC empirically shows that communicating with more peers can improve training quality [6]. With full model exchanges, however, each additional peer increases communication volume, motivating smaller messages that allow broader peer coverage at a lower communication cost Figure 2.

Figure 2. Communication patterns and outbound communication volume per worker for 16 replicas. GASLoC and SPICE support changing neighborhoods across communication rounds, illustrated at tt and t+1t+1, whereas Ring All-Reduce implements a global collective over a fixed ring. The square marks a reference worker, whose peer connections are emphasized. For a model with dd parameters, Ring All-Reduce transmits 1.875d1.875d values per worker, while dense two-peer GASLoC transmits 2d2d values. Instead, SPICE can send selected parameter values to two peers at 6.25%6.25\% density or four peers at 3.125%3.125\% density, both totaling qδd=0.125dq\delta d = 0.125d. SPICE combines dynamic neighborhoods with sparse messages, allowing a fixed value budget to be allocated between message density and peer count.

One approach to reducing communication is compression, through sparsification or quantization. Sparsification transmits only a subset of gradient or pseudogradient coordinates, selected, for example, through chunked Top-kk selection [38]. The resulting information loss can be mitigated through error feedback, which retains omitted update components for later communication [41]. Applying these mechanisms to parameter gossip is challenging because replicas evolve from different models: a compressed update relative to the sender’s model cannot generally be applied directly to the receiver’s model. Methods such as CHOCO-SGD address this mismatch by reconstructing communicated states from accumulated compressed updates on a fixed communication graph [18], but changing peers requires making the relevant reconstruction state available to new receivers. After a local update, sparsification lets a sender transmit selected parameter values directly, reducing message size while allowing any receiver to interpret them without a shared reference

¹Concordia University ²Mila ³Sorbonne Université ⁴CNRS.

Technical report. model. This motivates sparse parameter gossip that combines directly interpretable messages with the accumulated correction from error feedback.

The framework of Cagnasso et al. [6] provides a natural way to realize this design: it expresses gossip as a local pseudo-gradient formed from differences between a node’s parameters and those received from its peers. This formulation allows us to apply a sparse update with error feedback locally, then communicate the resulting parameter values at the selected coordinates. We use this mechanism to introduce SPICE, a decentralized training algorithm that combines chunked TOP-kk sparsification with error feedback while retaining the Nesterov outer optimizer for consensus acceleration. The transmitted parameter values incorporate accumulated error corrections and can be interpreted directly by any receiving peer, without persistent reconstruction caches.

Our contributions are:

  1. We propose SPICE, a decentralized training algorithm that integrates sparsification and error memory into parameter gossip through corrected absolute model endpoints.

  1. We empirically show that SPICE achieves lower validation loss than evaluated decentralized baselines, while transmitting less data (Figure 1).

Figure 1. Quality–communication tradeoff for decentralized LLM pretraining. Final validation loss for a 134M-parameter model trained on 2.70B tokens with 32 replicas. SPICE configurations vary peer count and sparsity. Outbound volume includes transmitted values and 12-bit packed indices and is normalized by the size of a dense model. Circled points and the dashed line indicate the empirical Pareto frontier among evaluated configurations. SPICE, achieving lower validation loss than all evaluated decentralized baselines, lies on the empirical Pareto frontier.

  1. We show that SPICE reduces end-to-end training time to a fixed token budget across several bandwidth caps compared with dense GASLoC and DDP, and retains most of the training throughput under transient bandwidth reductions, including severe slowdowns.

Related Works

Decentralized language-model training. Decentralized training replaces global aggregation with exchanges among selected peers, making the communication graph and peer selection rule central to information propagation [5, 31, 30]. D-PSGD combines neighbor averaging with stochastic optimization [23], while Stochastic Gradient Push and subsequent analyses extend decentralized training to directed or time-varying graphs and multiple local updates between exchanges [1] (Koloskova et al. 2020). For transformer training, DAdam studies local adaptive optimization with parameter mixing [47]. GASLoC builds on this approach for language-model pretraining by combining local adaptive updates, peer neighborhoods defined by a shared random permutation at each round, and an outer optimizer with momentum to accelerate consensus [6]. These parameter exchanges allow replicas with different models to aggregate their peers’ endpoints, but still transmit a full model to each peer. SPICE adopts the same neighborhood construction as GASLoC while reducing message size through sparse exchanges of corrected model endpoints.

Local optimization and compressed updates. Federated averaging and Local SGD reduce the overall communication volume by performing multiple local optimization steps between aggregation rounds [27, 40]. FedOpt generalizes model averaging by applying an outer optimizer to aggregated model differences used as pseudo-gradients [37]. DiLoCo applies this inner-outer optimization framework for language-model pretraining with local AdamW updates and an outer Nesterov optimizer [8]. GASLoC connects this framework to parameter gossip and recovers DiLoCo under exact global averaging [6]. Compression complements local updates by reducing the size of each message rather than the frequency of communication. Error feedback compensates for omitted information by retaining it for subsequent communication rounds [42, 16]. Beyond compensating for compression error, error feedback can be combined with explicit momentum: EF21-SGD establishes improved convergence guarantees by adding Polyak momentum to EF21 [9], while DeMo uses error feedback to compress communicated momentum in language-model training [36]. QSparse-local-SGD combines local updates with sparsification, quantization, and error compensation, but uses SGD without an outer momentum optimizer [2]. SparseLoCo studies this combination in language-model pretraining with local AdamW updates, using chunked TOP-kk compression of pseudo-gradients and error feedback as a surrogate for the outer momentum [38]. In SparseLoCo, replicas compute their pseudo-gradients from a shared round-start model, so averaging these displacements is equivalent to averaging their corresponding endpoints. In parameter gossip, replicas generally start from different models, so applying a peer’s displacement to the receiver’s model does not recover the intended peer’s endpoint. SPICE addresses this difference by transmitting selected absolute endpoint values that incorporate the accumulated error correction.

Compressed gossip. WASH communicates selected weights among models through random peer-to-peer permutations, keeping the models compatible with final weight averaging [10]. Compressed gossip methods instead combine neighboring models during training, with CHOCO-SGD reconstructing model estimates by accumulating compressed differences [18] (Koloskova* et al. 2020). Each message therefore depends on the receiver’s reconstruction history, so the standard formulation assumes a fixed communication graph to ensure that receivers retain the history needed. SPICE instead averages selected absolute endpoint values that incorporate error memory and require no receiver reconstruction history.

Communication topology. Beyond message compression, topology design controls the communication cost of propagating information among replicas. MATCHA samples graph matchings to balance communication cost and convergence [45], while one-peer exponential graphs and EquiTopo provide sparse communication schemes with favorable mixing properties [48, 39]. However, faster consensus alone does not establish better training quality, as the spectral gap can fail to predict decentralized learning performance under identically distributed data [44].

Method

SPICE

Sparse Parameter Gossip Challenges. Combining gossip with gradient compression and error feedback is challenging because these methods operate on model differences. It thus requires specifying the reference model of each transmitted update. Consider RR replicas with dd parameters and identical initializations. At round tt, replica ii starts from xi(t)∈Rdx_i^{(t)} \in\mathbb{R}^d, performs a local optimizer step to obtain yi(t)y_i^{(t)}, and computes its local displacement Δi(t)=xi(t)−yi(t)\Delta_i^{(t)} = x_i^{(t)} - y_i^{(t)}. For example, a local-step-then-mix form of decentralized SGD is

yi(t)=xi(t)−ηtgi(t),xi(t+1)=∑j=1Rwij(t)yj(t).(1)y_i^{(t)} = x_i^{(t)} - \eta_t g_i^{(t)}, \qquad x_i^{(t+1)} = \sum_{j=1}^{R} w_{ij}^{(t)} y_j^{(t)}. \tag*{(1)}

where gi(t)g_i^{(t)} is a local stochastic gradient, ηt\eta_t is the learning rate, and the nonnegative mixing weights sum to one and are nonzero only for the replica itself and its current peers (Koloskova et al., 2020). This exchange averages model parameters. Replicas generally start a round from different models, so applying a sender’s displacement Δj(t)\Delta_j^{(t)} to the receiver’s model xi(t)x_i^{(t)} does not recover yj(t)y_j^{(t)}. Compressing these displacements alone therefore does not directly reproduce parameter gossip.

CHOCO-SGD addresses this mismatch by maintaining a shared estimate x^j(t)\hat{x}_j^{(t)} of each sender’s model and compressing the difference from that estimate [18] (Koloskova* et al. 2020). Using the local-step-then-gossip ordering, its updates can be written as

mi(t)=C(yi(t)−x^i(t)),m_i^{(t)} = \mathcal{C}\left(y_i^{(t)}-\hat{x}_i^{(t)}\right),
x^i(t+1)=x^i(t)+mi(t),\hat{x}_i^{(t+1)} = \hat{x}_i^{(t)} + m_i^{(t)},
xi(t+1)=yi(t)+γ∑j=1Rwij(x^j(t+1)−x^i(t+1)),(2)x_i^{(t+1)} = y_i^{(t)} + \gamma\sum_{j=1}^{R} w_{ij}\left(\hat{x}_j^{(t+1)}-\hat{x}_i^{(t+1)}\right), \tag*{(2)}

where C\mathcal{C} is a compression operator, γ\gamma is the consensus step-size, and W=(wij)W=(w_{ij}) is a fixed symmetric doubly stochastic mixing matrix. The sender and its neighbors apply the same transmitted increment mi(t)m_i^{(t)} to maintain matching estimates. Information omitted by compression remains in the difference between the model and its estimate and can enter subsequent messages.

Its memory-efficient formulation avoids storing each neighbor estimate by maintaining their weighted sum si(t)=∑jwijx^j(t)s_i^{(t)}=\sum_j w_{ij}\hat{x}_j^{(t)}, requiring only three model-sized states. This formulation makes the dependence on a fixed graph explicit: the accumulated state sis_i encodes a specific weighted neighborhood, and changing that neighborhood generally requires information that cannot be recovered from the stored sum alone.

Sparse Parameters as Local Update Directions

Our proposed method SPICE, instead, transmits selected absolute parameter values, combining the endpoint exchange of GASLoC with sparsification and error feedback [6, 38]. To see the basic mechanism, first omit error feedback. When replica jj transmits coordinate yj(t)[c]y_j^{(t)}[c], receiver ii uses its own round-start parameter as the reference and forms

xi(t)[c]−yj(t)[c]=xi(t)[c]−xj(t)[c]⏟model disagreement+Δj(t)[c]⏟local displacement.(3)x_i^{(t)}[c]-y_j^{(t)}[c] = \underbrace{x_i^{(t)}[c]-x_j^{(t)}[c]}_{\text{model disagreement}} + \underbrace{\Delta_j^{(t)}[c]}_{\text{local displacement}}. \tag*{(3)}

This direction is relative to the receiver’s model and includes both the sender’s local displacement and the disagreement term needed for parameter gossip. As in GASLoC, averaging these differences over the available contributions gives a pseudo-gradient for an outer optimizer that provides consensus acceleration [6]. Absolute parameter values require neither a shared starting model nor a stored estimate of the sender’s previous model, allowing neighborhoods to change between rounds without transferring model estimates to new peers. As detailed below, this ability is central to combining a small communication budget with mixing across the replicas.

To complete this construction, we must specify which coordinates are sent, how their values incorporate retained error, and how to average messages with different supports.

Error-Feedback and Sparsification

We select coordinates using the local displacements and their retained history. Each replica maintains a residual buffer ei(t)e_i^{(t)}, initialized with ei(0)=0e_i^{(0)}=0. At round tt, it combines this buffer with the current local pseudo-gradient,

e~i(t)=βei(t)+Δi(t),(4)\tilde{e}_i^{(t)}=\beta e_i^{(t)}+\Delta_i^{(t)}, \tag*{(4)}

where β∈[0,1]\beta\in[0,1] controls the decay of retained history.

We partition the flattened accumulator into contiguous chunks of bb coordinates and select the kk entries with largest absolute values within each chunk. This chunked TOP-kk rule follows [38, 36] and assigns the same selection quota to each full chunk of a tensor, with density δ=k/b\delta=k/b. Let Mi(t)∈{0,1}d\mathcal{M}_i^{(t)}\in\{0,1\}^d denote the resulting mask and Ii(t)={c:Mi(t)[c]=1}\mathcal{I}_i^{(t)}=\{c:\mathcal{M}_i^{(t)}[c]=1\} the set of selected coordinates. The residual retained for the next round is

ei(t+1)=(1−Mi(t))⊙e~i(t),(5)e_i^{(t+1)}=\left(1-\mathcal{M}_i^{(t)}\right)\odot\tilde{e}_i^{(t)}, \tag*{(5)}

where ⊙\odot denotes elementwise multiplication.

Unselected components remain in the buffer and enter subsequent accumulators, allowing repeated displacements to accumulate rather than being discarded after each round. The mask determines which coordinates are transmitted, while Mi(t)⊙e~i(t)\mathcal{M}_i^{(t)}\odot\tilde{e}_i^{(t)} is the selected accumulated displacement used to construct the corrected endpoints below.

Correcting Endpoints with Error Feedback

On selected coordinates, the accumulator contains the current local displacement and the retained historical correction. Sending only yi(t)[Ii(t)]y_i^{(t)}[\mathcal{I}_i^{(t)}] would omit this correction. We therefore construct a corrected endpoint

zi(t)=yi(t)−Mi(t)⊙βei(t).(6)z_i^{(t)}=y_i^{(t)}-\mathcal{M}_i^{(t)}\odot\beta e_i^{(t)}. \tag*{(6)}

For a selected coordinate c∈Ii(t)c\in\mathcal{I}_i^{(t)}, this is equivalent to

zi(t)[c]=xi(t)[c]−e~i(t)[c].(7)z_i^{(t)}[c]=x_i^{(t)}[c]-\tilde{e}_i^{(t)}[c]. \tag*{(7)}

Replica ii sends (Ii(t),zi(t)[Ii(t)])(\mathcal{I}_i^{(t)},z_i^{(t)}[\mathcal{I}_i^{(t)}]) to its peers. On unselected coordinates, zi(t)[c]=yi(t)[c]z_i^{(t)}[c]=y_i^{(t)}[c], so the self endpoint retains the current local update.

The accumulated displacement is encoded as xi(t)[c]−zi(t)[c]x_i^{(t)}[c]-z_i^{(t)}[c] and cleared from the residual on selected coordinates, while remaining in the buffer on unselected coordinates. Together, these two cases give

e~i(t)=βei(t)+Δi(t)=ei(t+1)⏟retained in memory+Mi(t)⊙(xi(t)−zi(t))⏟encoded in transmitted endpoints(8)\tilde{e}_i^{(t)} = \beta e_i^{(t)} + \Delta_i^{(t)} = \underbrace{e_i^{(t+1)}}_{\text{retained in memory}} + \underbrace{M_i^{(t)} \odot\left(x_i^{(t)} - z_i^{(t)}\right)}_{\text{encoded in transmitted endpoints}} \tag*{(8)}

Thus, any historical correction cleared from memory is incorporated into the transmitted endpoint values. We next combine the full self endpoint with the selected peer values through coordinate-wise averaging.

Randomized Coordinate-Wise Gossip and Outer Optimization

In methods such as CHOCO-SGD, messages depend on accumulated reconstruction state, which must be available when introducing a new peer. The corrected absolute endpoints in SPICE can instead be interpreted by any receiver without previous exchanges. This allows peer neighborhoods to change between rounds. We use randomized peer selection, motivated by analyses showing that randomized one- and two-peer exchanges can mix efficiently as the number of replicas grows [6]. At each round, a newly sampled shared permutation determines the neighborhoods, and each replica sends its selected endpoint values to qq peers.

Each message contains a fraction δ\delta of the model coordinates, giving qδdq\delta d transmitted parameter values per round (Figure 2). CHOCO-SGD can match this payload using a fixed graph with the same peer count and density. However, sparse fixed topologies such as rings can limit mixing and hinder training as the number of replicas grows [32]. The advantage of SPICE is that it combines sparse messages with changing communication partners without transferring reconstruction state. Relative to the complete-graph CHOCO-SGD baseline used in our experiments, contacting only qq peers reduces outbound volume by a factor of (R−1)/q(R-1)/q at equal message density.

Since peers can select different coordinates, the contributors to coordinate cc are

Pi,c(t)={i}∪{j∈Ni(t):c∈Ij(t)}.(9)\mathcal{P}_{i,c}^{(t)} = \{i\} \cup\{j \in\mathcal{N}_i^{(t)} : c \in\mathcal{I}_j^{(t)}\}. \tag*{(9)}

We average their endpoint values and form the outer residual,

Gi(t)[c]=xi(t)[c]−1∣Pi,c(t)∣∑j∈Pi,c(t)zj(t)[c].(10)G_i^{(t)}[c] = x_i^{(t)}[c] - \frac{1}{\lvert\mathcal{P}_{i,c}^{(t)} \rvert}\sum_{j \in\mathcal{P}_{i,c}^{(t)}} z_j^{(t)}[c]. \tag*{(10)}

The self endpoint always contributes, including coordinates outside Ii(t)\mathcal{I}_i^{(t)}. Peers that do not transmit a coordinate contribute neither a value nor a count to its average. If no peer transmits cc, then Gi(t)[c]=xi(t)[c]−zi(t)[c]G_i^{(t)}[c] = x_i^{(t)}[c] - z_i^{(t)}[c], and the outer residual retains the local displacement and any selected historical correction.

To separate the effects of local optimization and model disagreement, one can define the displacement Dj(t)=xj(t)−zj(t)=Δj(t)+Mj(t)⊙βej(t)D_j^{(t)} = x_j^{(t)} - z_j^{(t)} = \Delta_j^{(t)} + M_j^{(t)} \odot\beta e_j^{(t)}. The outer residual can then be written as

Gi(t)[c]=xi(t)[c]−∑j∈Pi,c(t)xj(t)[c]∣Pi,c(t)∣⏟consensus disagreement+∑j∈Pi,c(t)Dj(t)[c]∣Pi,c(t)∣⏟local displacements.(11)G_i^{(t)}[c] = x_i^{(t)}[c] - \underbrace{\frac{\sum_{j \in\mathcal{P}_{i,c}^{(t)}} x_j^{(t)}[c]}{\lvert\mathcal{P}_{i,c}^{(t)} \rvert}}_{\text{consensus disagreement}} + \underbrace{\frac{\sum_{j \in\mathcal{P}_{i,c}^{(t)}} D_j^{(t)}[c]}{\lvert\mathcal{P}_{i,c}^{(t)} \rvert}}_{\text{local displacements}}. \tag*{(11)}

The first term corrects disagreement between the receiver’s starting model and the neighbors’ starting models. The second averages their local displacements and selected historical corrections. When the contributing replicas share a starting model, the disagreement term vanishes and gossip reduces to averaging these corrected displacements. Otherwise, coordinate selection affects both displacement aggregation and consensus.

Following GASLoC [6], we apply an outer Nesterov momentum optimizer to Gi(t)G_i^{(t)}, starting from xi(t)x_i^{(t)}. This choice follows the connection between outer momentum and accelerated consensus developed for dense gossip by Cagnasso et al. [6]. Algorithm 1 summarizes this training procedure.

With full support (Top-k=bk = b) and zero initial residual, the residual remains zero and zi(t)=yi(t)z_i^{(t)} = y_i^{(t)} at every round, recovering dense GASLoC.

Communication Cost

Each worker saves its round-start parameters to compute the displacement used for support selection and the pseudo-gradient used by the outer optimizer. For selection, the trainable local parameter shards are viewed as one vector and partitioned into chunks of bb coordinates. Chunks can cross PyTorch’s parameter boundaries, so participating workers have to use a consistent parameter ordering and shard layout.

The selected coordinates become a message containing two buffers: their indices and their parameter values. For b≤4096b \leq4096, an index needs only 12 bits to identify a position within its chunk; thus, two indices can be packed into three bytes. Since each chunk contributes the Top-kk entries in order, the receiver infers the chunk from the entry’s position in the message.

Given this message format, let dlocd_{\mathrm{loc}} be the number of trainable parameters in a local shard and qq the number of peers. Assuming complete chunks and packed index pairs, each

Algorithm 1 SPICE

1: Input: initialize x1(0)=⋯=xR(0)x_1^{(0)}=\cdots=x_R^{(0)}, initialize ei(0)=0 ∀ie_i^{(0)}=0\ \forall i, Optimizerin_{\mathrm{in}}, Optimizerout_{\mathrm{out}}, EF memory decay β\beta, and qq, kk, bb

2: for t=0,1,…,T−1t=0,1,\ldots,T-1 do

3: Sample a shared permutation σ(t)\sigma^{(t)} of {1,…,R}\{1,\ldots,R\}

4: for Worker i∈{1,…,R}i\in\{1,\ldots,R\} in parallel do

    ▹ Inner update

5: yi(t)←Optimizer⁡in(xi(t),∇f(xi(t)))y_i^{(t)}\leftarrow\operatorname{Optimizer}_{\mathrm{in}}\left(x_i^{(t)},\nabla f(x_i^{(t)})\right)

6: Δi(t)←xi(t)−yi(t)\Delta_i^{(t)}\leftarrow x_i^{(t)}-y_i^{(t)}

    ▹ Error-feedback support selection

7: e~i(t)←βei(t)+Δi(t)\widetilde{e}_i^{(t)}\leftarrow\beta e_i^{(t)}+\Delta_i^{(t)}

8: Mi(t),Ii(t)←Top-k⁡(e~i(t))\mathcal{M}_i^{(t)},\mathcal{I}_i^{(t)}\leftarrow\operatorname{Top\text{-}k}\left(\widetilde{e}_i^{(t)}\right)

9: zi(t)←yi(t)−Mi(t)⊙βei(t)z_i^{(t)}\leftarrow y_i^{(t)}-\mathcal{M}_i^{(t)}\odot\beta e_i^{(t)}

10: ei(t+1)←(1−Mi(t))⊙e~i(t)e_i^{(t+1)}\leftarrow\left(1-\mathcal{M}_i^{(t)}\right)\odot\widetilde{e}_i^{(t)}

    ▹ Gossip step

11: Ni(t)←SAMPLE_PEERS⁡(σ(t),i)\mathcal{N}_i^{(t)}\leftarrow\operatorname{SAMPLE\_PEERS}\left(\sigma^{(t)},i\right)

12: exchange⁡(Ii(t),zi(t)[Ii(t)])\operatorname{exchange}\left(\mathcal{I}_i^{(t)},z_i^{(t)}[\mathcal{I}_i^{(t)}]\right) with all j∈Ni(t)j\in\mathcal{N}_i^{(t)}

13: for c∈{1,…,d}c\in\{1,\ldots,d\} do

14: Pi,c(t)←{i}∪{j∈Ni(t):c∈Ij(t)}\mathcal{P}_{i,c}^{(t)}\leftarrow\{i\}\cup\{j\in\mathcal{N}_i^{(t)}:c\in\mathcal{I}_j^{(t)}\}

15: Gi(t)[c]←xi(t)[c]−1∣Pi,c(t)∣∑j∈Pi,c(t)zj(t)[c]G_i^{(t)}[c]\leftarrow x_i^{(t)}[c]-\frac{1}{|\mathcal{P}_{i,c}^{(t)}|}\sum_{j\in\mathcal{P}_{i,c}^{(t)}}z_j^{(t)}[c]

16: end for

    ▹ Outer update

17: xi(t+1)←Optimizer⁡out(xi(t),Gi(t))x_i^{(t+1)}\leftarrow\operatorname{Optimizer}_{\mathrm{out}}\left(x_i^{(t)},G_i^{(t)}\right)

18: end for

19: end for

20: Output: xˉ(T)=1R∑i=1Rxi(T)\bar{x}^{(T)}=\frac{1}{R}\sum_{i=1}^{R}x_i^{(T)}

worker selects s=(k/b)dloc=δdlocs=(k/b)d_{\mathrm{loc}}=\delta d_{\mathrm{loc}} coordinates and sends the same message to its qq peers. If each value occupies vv bytes and the indices add 1.5 bytes per selected coordinate, then

Vsend=qδdloc(v+1.5).(12)V_{\mathrm{send}}=q\delta d_{\mathrm{loc}}(v+1.5). \tag*{(12)}

Dense exchange with the same peers sends qvdlocqvd_{\mathrm{loc}} bytes, so the payload ratio is δ(1+3/(2v))\delta(1+3/(2v)).

Experimental Results

Experimental Setup

We train two dense Llama-3-style decoder-only Transformers (Grattafiori et al., 2024) with 134M and 551M parameters, and the mixture-of-experts model OLMoE1B-7B (Muennighoff et al., [29]), on FineWeb (Penedo et al., [35]). Their training budgets are 2.70B, 11.01B, and 26.05B tokens, respectively, corresponding to approximately 20 tokens per parameter for the dense models (Hoffmann et al., [12]) and per active parameter for OLMoE. The data are tokenized with the LLaMA-2 tokenizer (Touvron et al., [43]) and packed into sequences of 2048 tokens, with a global batch of 1024 sequences, or approximately 2M tokens, shared across replicas. All methods perform local

updates with AdamW, with the peak learning rate tuned separately for each method and a schedule consisting of warmup followed by cosine decay. Its second-order momentum β2\beta_2 is scaled according to Belilovsky et al. [3]. Where applicable, we also tune the outer learning rate and Nesterov momentum.

Baselines. We compare SPICE with AdamW DDP, SparseLoCo, DAdam, GASLoC, and CHOCO-SGD, with replicas communicating after every local optimizer update in all decentralized methods. AdamW DDP provides the reference for training with exact gradient aggregation at every update (Li et al., 2020). We choose DAdam as an adaptive decentralized baseline evaluated on transformer training ([47]). We also compare with GASLoC using one or two peers per round, which provides the dense counterpart of SPICE and measures the quality and communication cost achievable through peer selection alone ([6]). Finally, CHOCO-SGD provides a baseline for compressed decentralized communication ([19]). We evaluate it on the complete graph because it relies on a static communication graph, a dependence explicit in its memory-efficient formulation. Fixed sparse graphs could match the peer count of our randomized selections (e.g., a ring for q=2q=2), but would impose a substantial mixing disadvantage, potentially slowing convergence ([32]). We evaluate SparseLoCo at the same sparsity density as the other sparse baselines, without quantization or outer momentum, and with H=1H=1. Table 1 summarizes the key distinctions among these methods.

MethodP2PAccelerationSparseDynamic Neighborhoods
AdamW DDP✗✗✗✗
DAdam✓✗✗✓
GASLoC✓✓✗✓
SparseLoCo✗✗✓✗
CHOCO-SGD✓✗✓✗
SPICE✓✓✓✓

Table 1. Comparison of communication and optimization capabilities. Methods comparison: ✓ and ✗ indicate supported and unsupported features, respectively. SPICE brings together peer-to-peer communication, communication acceleration, sparse communication, and dynamic neighborhoods in a unified framework.

Implementation. We evaluate training quality with a simulator of local updates and message exchanges, allowing us to study up to 128 replicas without requiring 128 physical GPUs. We measure execution timings separately using a TorchTitan implementation (Liang et al., [24]). This implementation packs the selected indices within each chunk into 12 bits each, storing two indices in three bytes, and exchanges the packed indices and selected parameter values through actual peer-to-peer communication. Values retain the parameter dtype and are exchanged in batches after the packed indices are sent. The receiver then unpacks the indices and accumulates the received values at the corresponding coordinates.

MethodPeers qqR=8R = 8R=16R = 16R=32R = 32
AdamW DDP (ref.)gc2.632.632.63
DAdamR−1R - 12.702.732.77
GASLoC12.672.712.75
GASLoC22.642.682.73
SparseLoCogc2.712.742.78
CHOCO-SGDR−1R - 12.732.812.95
SPICE (ours)R/2R/22.652.682.70

Table 2b. (b) 551M parameters, 11.01B tokens.

Training Results

For the 134M model, SPICE achieves the best validation loss among both the sparse and decentralized methods evaluated in Table 2a. Increasing the replica count from 8 to 128 raises validation loss by only 1.3% for SPICE, compared with 5.6% for dense two-peer GASLoC. This reduces a known difficulty of methods using inner and outer optimizers, whose performance can degrade as the replica count increases ([13, 3]). Each message contains only 3.125% of the parameter coordinates, transmitting more than 30 times fewer parameter values than a dense decentralized exchange. This sparsity allows us to expand the neighborhood proportionally to the replica count, using q=R/2q = R/2, at a fraction of the volume required for dense exchanges with the same peers. As the fixed global batch is divided among more replicas, the neighborhood grows by the same factor that the local batch size decreases, keeping the total number of tokens processed by the communicating peers constant. For each coordinate, however, only peers that select it contribute to the outer gradient, so this does not imply constant token coverage at every coordinate. The token coverage maintained across the neighborhood provides a possible explanation for the reduced degradation, which results in a validation loss advantage of 0.14 over dense two-peer GASLoC at 128 replicas.

MethodPeers qqR=8R = 8R=16R = 16R=32R = 32R=64R = 64R=128R = 128
AdamW DDP (ref.)gc3.173.173.173.173.17
DAdamR−1R - 13.233.253.303.363.42
GASLoC13.203.233.283.353.44
GASLoC23.193.213.243.313.37
SparseLoCogc3.273.283.323.383.42
CHOCO-SGDR−1R - 13.273.353.473.673.86
SPICE (ours)R/2R/23.193.193.203.213.23

Table 2. Final validation loss. (a) Validation loss for the 134M model trained on 2.70B tokens with RR replicas. (b) Validation loss for the 551M model trained on 11.01B tokens across RR replicas. DDP and SparseLoCo communicate through global collectives (gc), while the other methods use peer-to-peer communication. DAdam and CHOCO-SGD use a complete communication graph, while GASLoC and SPICE sample a time-varying graph. Methods with sparse communication use density δ=3.125%\delta= 3.125\%, with k=128k = 128 and block size b=4096b = 4096. We highlight in bold the best decentralized result for each replica count and underline the best result among sparse methods. In case of a tie, the method with the lowest communication volume is highlighted. SPICE’s advantage over the evaluated decentralized baselines grows with the replica count, as its expanding neighborhood limits loss degradation.

Using the same density and neighborhood rule, SPICE retains the best validation loss among the evaluated sparse methods when the model size increases from 134M to 551M parameters, as shown in Table 2b. As with the 134M model, its performance is comparable to dense two-peer GASLoC at smaller replica counts, with an increasing advantage as the replica count grows. From 8 to 32 replicas, its validation loss increases by 1.9%, compared with 3.4% for the dense variant, extending the reduced degradation to a larger model and training budget.

Muenighoff et al. (2025) Table 3 extends the evaluation to OLMoE-1B-7B, a mixture-of-experts model with 7B parameters. With 8 replicas, SPICE exchanges 6.25% of the parameter coordinates with four peers per round, using one-eighth the communication volume than dense two-peer GASLoC. It achieves a nearly identical validation loss and comparable accuracy in HellaSwag ([49]) and PIQA ([4]), with both accuracy differences falling within the reported error bars. These results show that the communication savings also extend to different and larger model architectures while preserving competitive training quality.

MethodComm. Vol.LossHSwagPIQA
AdamW DDP (ref.)1.75d1.75d2.36252.71±0.4952.71 \pm0.4972.36±1.0472.36 \pm1.04
DAdam-complete7d7d2.42148.87±0.4948.87 \pm0.4971.71±1.0571.71 \pm1.05
GASLoC (q=1q = 1)dd2.37051.93±0.5051.93 \pm0.5071.60±1.0471.60 \pm1.04
GASLoC (q=2q = 2)2d2d2.36752.56±0.5052.56 \pm0.5072.58±1.0572.58 \pm1.05
SPICE (ours) (q=4q = 4)0.25d0.25d2.36852.38±0.5052.38 \pm0.5072.42±1.0572.42 \pm1.05

Table 3. 7B Mixture of Experts. Final validation loss and downstream length-normalized accuracy for the 7B MoE trained on 26B tokens across 8 replicas. SPICE uses TOP-k=256k = 256 and block size b=4096b = 4096 for a density δ=6.25%\delta= 6.25\%. Compared with dense two-peer GASLoC, SPICE achieves comparable model quality with 8× less communication volume.

Effect of Density and Peer Count

Peer count qq and density δ\delta are the two main controls of communication volume in SPICE. Their product qδq\delta determines the total number of parameter values sent per round relative to one dense model exchange, with the corresponding byte cost given in § 3.6. The same communication volume can therefore be allocated to more peers or more coordinates per peer. Figure 3a compares how these choices affect validation loss.

Figure 3. Effect of density and peer count on validation loss. The 134M model is trained with 32 replicas for 2.70B tokens. (a) shows the joint effect of density and peer count, the two factors that determine the total communication volume. (b) compares validation loss at fixed outbound communication volume, showing the trade-off between more peers and more coordinates per peer under a fixed budget. Increasing density brings diminishing improvements when more peers are used, while configurations with fewer peers remain more sensitive to sparsification. At small communication budgets, sending sparse messages to more peers yields lower validation losses.

With fewer peers, increasing density reduces validation loss throughout the tested range. With more peers, loss instead levels off once density reaches a few percent, and the advantage of larger neighborhoods decreases as density increases. Larger neighborhoods are therefore most beneficial under aggressive sparsification. This is consistent with sparse messages from more peers partly compensating for the limited number of coordinates in each message. The resulting exchanges may already provide enough information that transmitting additional coordinates brings little improvement, even while communication volume remains below that of a dense model exchange.

Figure 3b compares these configurations at equal outbound communication volume, including the packed indices. At smaller communication budgets, allocating the same volume to more peers reduces validation loss. With sixteen peers, SPICE improves on both dense GASLoC baselines while sending less than half the bytes required for one dense model exchange per round.

Training Under Limited Bandwidth

Training over the internet often relies on connections with substantially less bandwidth than datacenter interconnects ([46, 14, 25]). In this setting, communication can limit training speed, while bandwidth stragglers and temporary link failures can further delay synchronization. These constraints motivate measuring how reducing communication volume affects training time.

End-to-End Training Time

We benchmark the 134M and 551M models on eight NVIDIA RTX A6000 GPUs within a single machine, with one training replica per GPU, using our TorchTitan implementation ([24]) with actual peer-to-peer communication. For SPICE, each exchange transmits the selected parameter values and their 12-bit packed indices. The timings therefore include index packing, transmission of both indices and values, and unpacking and accumulation at the receiver. To control bandwidth, we place each replica in a separate Linux network namespace and connect the namespaces through a virtual Ethernet bridge. We use tc netem to cap outgoing traffic at the replica interface and incoming traffic at the corresponding interface on the host. The specified rate therefore limits each replica’s aggregate sending and receiving bandwidth separately, and is shared by all its peer exchanges. We apply the same rate to all eight replicas and evaluate caps of 1, 2, 5, and 10 Gbps. NCCL uses its socket transport over these interfaces, with direct GPU transfers, shared memory transport, and InfiniBand disabled to prevent communication from bypassing the caps. These runs impose bandwidth limits without injecting additional latency or link failures. The uncapped setting instead uses NCCL without the namespace bandwidth controls.

Figure 4 shows that SPICE achieves the highest training throughput for both model sizes at every tested bandwidth. Its throughput changes little as bandwidth decreases from 10 to 1 Gbps, while all dense methods slow down over this range, increasing the advantage of sparsification under tighter bandwidth limits. This advantage holds even though SPICE communicates with four peers, compared with one or two for dense GASLoC. With TOP-k=128k = 128 and b=4096b = 4096, each sparse message contains only 3.125% of the model coordinates, allowing SPICE to reach more peers while sending less data overall, including the coordinate indices. The resulting throughput shows that sparsification can support a larger peer count without incurring the communication cost of dense exchanges.

Figure 4. Training throughput under limited bandwidth. Aggregate throughput in thousands of tokens per second for the 134M and 551M models on eight RTX A6000 GPUs, with the same bandwidth cap applied separately to each replica’s links. Higher is better. SPICE uses four peers, TOP-k=128k = 128, and block size b=4096b = 4096, corresponding to density δ=3.125%\delta= 3.125\%. SPICE achieves the highest throughput at every tested bandwidth and retains nearly constant throughput as the cap decreases from 10 to 1 Gbps, while DDP and both dense GASLoC variants lose throughput over this range.

To assess how this throughput advantage affects training quality, we use the time required by DDP to complete its training budget as the reference. At each bandwidth cap, we train SPICE for more steps to match this elapsed time, allowing it to process more tokens. Figure 5 shows that SPICE achieves lower validation loss on the 134M model under this comparison, with a reduction of 6% at 1 Gbps. The additional steps enabled by sparse communication therefore improve validation loss within a fixed time window.

Figure 5. Time-matched training under limited bandwidth. Validation loss differences relative to DDP for SPICE (q=4q = 4, δ=3.125%\delta= 3.125\%), training the 134M model on 8 A6000 GPUs with uniform bandwidth caps set using τ\tau. At equal training time, SPICE outperforms DDP at all bandwidth caps shown with a reduction of 6% in the slowest setting.

Communication Stragglers Between Islands

Geographically distributed training resources can form communication islands, with fast connections within sites and slower internet connections between them ([14, 25]). The connections between islands can also vary over time, with Internet measurements reporting substantial throughput degradation over the course of a day ([28]). A possible disruption is a temporary bandwidth reduction on the paths connecting two islands, while connections within each island and paths involving other islands remain available. When several exchanges share the affected network infrastructure, the slowdown can affect multiple worker pairs, so changing peer neighborhoods can repeatedly encounter the same bottleneck.

In this setting, workers remain operational and can compute locally, but exchanges over affected paths take substantially longer, becoming communication stragglers. A link failure is an extreme case in which no message can traverse the affected link until connectivity is restored. Such a failure does not necessarily isolate an island, since information may still propagate through other islands when alternative paths remain available.

We emulate this scenario on the same eight-GPU setup, assigning the replicas to three logical islands containing three, three, and two replicas (Figure 6a). Using the network namespaces described above, we reduce bandwidth in both directions between the two larger islands while leaving all other paths unchanged. We test a cap of 500 Mbps and a more severe cap of 100 Mbps, applied separately to each directed worker pair between these islands rather than shared by the whole island. Each run includes an initial healthy phase, a period of reduced bandwidth, and a final phase with the original bandwidth restored. Workers and their control connections remain operational throughout, isolating the effect of degraded payload paths from worker failures.

Island topology and training throughput under slowed links

Figure 6. Training throughput with slow links between communication islands. Eight replicas of the 134M model run on eight NVIDIA RTX A6000 GPUs, with one replica per GPU. (a) Replicas are grouped into islands A, B, and C containing three, three, and two replicas, respectively. Dashed edges mark the affected links between A and B. (b, c) Throughput when each directed worker pair between A and B is capped from minute 5 to minute 10 at 500 Mbps and 100 Mbps, respectively. All other paths remain unchanged, and the original bandwidth is restored at minute 10. Throughput is measured in 30 s bins and normalized by each method’s top throughput during the initial healthy phase. SPICE limits throughput loss at both bandwidth caps. Windowed SPICE is additionally evaluated at 100 Mbps, where bounded waiting further improves throughput.

We compare SPICE against DDP managed by TorchFT, allowing the collective baseline to use its recovery mechanisms. At 500 Mbps, smaller messages already limit the effect of slow links: SPICE retains most of its healthy throughput, while DDP loses more than two thirds of it (Figure 6b). Further reducing the bandwidth to 100 Mbps prevents DDP from committing most of the updates during the 5-minute slowdown, while SPICE retains approximately two thirds of its healthy throughput (Figure 6c). Sparsification alone allows exchanges to complete faster despite the reduced bandwidth, because each message sends less than a twentieth of the full payload.

Improving throughput with bounded waiting

The results above wait for all requested peer contributions, but peer-to-peer exchanges allow each replica to aggregate available contributions without waiting for every requested peer. To exploit this flexibility under communication stragglers, one could use a windowed approach to gossip, which combines background reception with a bounded communication window. Each replica posts a receive operation in the background so that peers can send updates before it reaches its own exchange stage. At the deadline, it aggregates only complete peer contributions and proceeds with its outer update, using only its own endpoint if none arrived in time, as it would do for coordinates that were not shared by any other peer. A global collective instead requires the prescribed participation to complete its result, even when communication overlaps with computation.

Sparsification allows each replica to contact more peers under the same communication budget, increasing the opportunities to receive contributions before the deadline. This benefit depends on reaching peers through unaffected paths, since peers sharing a bottleneck can be delayed together. We therefore evaluate SPICE with windowed gossip in the same island setup. During the initial healthy period, we measure the distribution of complete sparse exchange times. The communication window is fixed at 2×2 \times the measured 99th percentile and kept unchanged during the period of reduced bandwidth to accommodate small differences in replica timing. Figure 6c shows that windowed gossip further improves throughput under the more severe slowdown by allowing updates to proceed with fewer contributions. With this window, committed updates used an average of 2.48 ±\pm 0.88 peer contributions out of four requested peers, while retaining approximately 86% of healthy throughput.

Conclusion

We introduced SPICE, a decentralized training method that combines sparse parameter exchanges with error feedback to make broader peer neighborhoods affordable. Under tight communication budgets, exchanging fewer coordinates with more peers improves training quality over denser exchanges with fewer peers. This design can outperform dense gossip while communicating less and limiting quality degradation as the replica count grows. Our implementation turns these smaller exchanges into higher training throughput under bandwidth constraints and maintains much of this performance during transient link slowdowns. These results support decentralized language model pretraining over bandwidth-constrained networks.

Acknowledgments

This work was supported by PEPR IA on grant SHARP ANR-23-PEIA-0008 and PEPR NUMPEX on grant DAIMOS ANR-25-EXNU-0002. Part of this work was granted access to the JeanZay HPC/AI resources of IDRIS under the allocation AD011015884R1, AD011017481, AD011016641, AD011017661 and AD011017766. E.B. and P.C. acknowledge funding from NSERC Discovery and FRQNT New Scholar.

B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock,

B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Inan, H., Shojanazerí, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj, I., Molbyog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A., L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P., Dong, N., Cheng, N., Cherngouz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X.,

AI Usage Disclosure

Claude and ChatGPT assisted with manuscript editing to improve clarity and grammar. ChatGPT also assisted with the implementation of software functions and reviewed code for potential errors. These tools were not used to formulate the research questions, develop the proposed method, design the experiments, or interpret the empirical findings. The authors reviewed and verified all AI-assisted text and code and take responsibility for the final content.

Additional Results

Tables 4 and 5 extend Table 2 with validation losses for additional peer counts, providing the numerical results for the analysis of scaling across replicas. Table 6 supplements Figure 4 with absolute training times and additional combinations of peer count and message density.

Expanding neighborhoods reduces degradation with replica count

We examine how the number of contacted peers affects training quality as the replica count increases. Tables 4 and 5 report validation losses across replica and peer counts at a fixed message density of 3.125%. For the 134M model, increasing the replica count from 8 to 128 while keeping four peers raises validation loss by 10.3%. Expanding the neighborhood according to q=R/2q = R/2 instead limits this increase to 1.3%. At fixed message density, expanding the neighborhood therefore reduces the degradation observed when training is distributed across more replicas.

Figure 7 groups configurations by their peer batch size, qB/RqB/R, where B=1024B = 1024 sequences is the fixed global batch. Along each curve, the peer count grows proportionally to the replica count, keeping the total batch processed by the contacted peers constant while each replica’s local batch shrinks. Validation loss remains approximately stable across these configurations, with larger peer batches consistently yielding lower loss. These results are consistent with expanding neighborhoods compensating for the shrinking local batch, although they do not isolate the mechanism responsible. In particular, peer batch size is not an effective optimization batch: peers update different local models, and only those transmitting a given coordinate contribute to its aggregation.

Figure 7. Figure 7. Validation loss across replica counts at fixed peer batch size. The 134M model is trained on 2.70B tokens at density δ=3.125%\delta= 3.125\%. Each curve keeps the peer batch size qB/RqB/R constant, where qq is the number of contacted peers and B=1024B = 1024 sequences is the global batch. Peer batch size excludes the replica’s own batch. Expanding the neighborhood proportionally to replica count limits validation-loss degradation.

At peer batch sizes of 32 and 64 sequences, validation loss decreases slightly as the neighborhood expands. This suggests that peer batch size alone does not fully determine quality. The trend is consistent with Figure 3, where allocating a fixed communication volume to more peers with sparser messages yields lower validation loss.

The 551M results in Table 5 confirm at a larger model scale that expanding the neighborhood reduces validation-loss degradation as the replica count increases.

MethodPeers qqComm. VolumeP2PSparseR=8R = 8R=16R = 16R=32R = 32
AdamW DDP (ref.)gc2(R−1)/R d≈2d2(R-1)/R\,d \approx2d✗✗2.63
DAdamR−1R - 1(R−1) d(R - 1)\,d✓✗2.702.732.77
GASLoC1dd✓✗2.672.712.75
GASLoC22d2d✓✗2.642.682.73
SparseLoCogc(R−1)δ d(R - 1)\delta\,d✗✓2.712.742.78
CHOCO-SGDR−1R - 1(R−1)δ d(R - 1)\delta\,d✓✓2.732.812.95
SPICE22δ d=0.06d2\delta\,d = 0.06d✓✓2.712.752.77
44δ d=0.12d4\delta\,d = 0.12d✓✓<u>2.65</u>2.712.74
88δ d=0.25d8\delta\,d = 0.25d✓✓—2.682.71
1616δ d=0.5d16\delta\,d = 0.5d✓✓——2.70

Table 5. 551M model validation loss. Final validation loss after training on 11.01B tokens, varying the number of replicas and selected peers at fixed density δ=3.125%\delta= 3.125\%, with TOP-k=128k = 128 and block size b=4096b = 4096. For each replica count, bold marks the best decentralized result and underlining marks the best sparse result. These results confirm at a larger model scale the diagonal trend observed for SPICE at 134M.

Reproducibility

Peer sampling

The SAMPLE_PEERS procedure in Algorithm 1 is implemented as follows. At each round tt, every replica independently generates the same uniform permutation σ(t)\sigma^{(t)} of the RR replicas using a shared seed and the round index tt. This permutation arranges replicas on a cycle without communication to coordinate peer selection. For an even peer count q<Rq < R, each replica exchanges messages with the q/2q/2 replicas preceding it and the q/2q/2 replicas following it, with positions taken modulo RR. For the one-peer GASLoC configuration, RR must be even, and consecutive replicas in the permutation form disjoint pairs: the first exchanges with the second, the third with the fourth, and so on. Both constructions give each replica exactly qq distinct peers and make every connection symmetric. Generating a fresh permutation at each round changes the communication partners while preserving the peer count, allowing information to propagate beyond a fixed neighborhood.

Software and hardware setup

The training simulator uses Python 3.12, PyTorch 2.10.0, and TorchTitan 0.2.1. The implementation used for timing measurements is based on TorchTitan 0.2.2 and the June 23, 2026 nightly release of torchft. All models use the LLaMA-2 tokenizer with a vocabulary of 32K tokens, available from the Together AI repository.

Training runs for the 134M and 551M models use compute nodes with four NVIDIA H100 GPUs per node, connected through NVLink within each node and InfiniBand across nodes. These runs simulate multiple logical replicas on each GPU. OLMoE training runs use a single node with eight NVIDIA H200 GPUs connected through NVLink. Training times and throughput are measured separately on a single node with eight NVIDIA RTX A6000 GPUs, with one physical training replica per GPU.

MethodPeers qqComm. VolumeP2PSparseR=8R = 8R=16R = 16R=32R = 32R=64R = 64R=128R = 128
AdamW DDP (ref.)gc2(R−1)/R d≈2d2(R-1)/R\,d \approx2d✗✗3.17
DAdamR−1R - 1(R−1) d(R - 1)\,d✓✗3.233.253.303.363.42
GASLoC1dd✓✗3.203.233.283.353.44
GASLoC22d2d✓✗3.193.213.243.313.37
SparseLoCogc(R−1)δ d(R - 1)\delta\,d✗✓3.273.283.323.383.42
CHOCO-SGDR−1R - 1(R−1)δ d(R - 1)\delta\,d✓✓3.273.353.473.673.86
SPICE22δ d=0.06d2\delta\,d = 0.06d✓✓3.233.303.403.543.73
44δ d=0.12d4\delta\,d = 0.12d✓✓3.193.233.313.393.52
88δ d=0.25d8\delta\,d = 0.25d✓✓—3.193.243.303.39
1616δ d=0.5d16\delta\,d = 0.5d✓✓——3.203.233.31
3232δ d=d32\delta\,d = d✓✓———3.213.25
6464δ d=2d64\delta\,d = 2d✓✓————3.23

Table 4. 134M model validation loss. Final validation loss after training on 2.70B tokens, varying the number of replicas and selected peers at fixed density δ=3.125%\delta= 3.125\%, with TOP-k=128k = 128 and block size b=4096b = 4096. For each replica count, bold marks the best decentralized result and underlining marks the best sparse result. For SPICE, the diagonals trace paths of low validation loss degradation as the replica count increases.

Communication bandwidth limits

For the global bandwidth experiments, each of the eight GPU replicas runs in a separate Linux network namespace connected through a virtual Ethernet pair to a private bridge. Linux traffic control tc applies the same fixed bandwidth cap to every replica, independently limiting its aggregate outgoing and incoming traffic. The outgoing cap is installed on the namespace interface, and the incoming cap on the host end of its virtual Ethernet pair, using netem with a queue limit of 10,000 packets and packet offloads disabled. All peer connections of a replica share these caps. NCCL uses IPv4 sockets over the virtual interface, with any other GPU transport technology disabled to prevent communication from bypassing the limits. Control and monitoring traffic also share the capped interfaces.

For the selective bandwidth experiment, we compare DDP and SPICE using the same namespace isolation and assign replicas 1–3 to island A, 4–6 to island B, and 7–8 to island C (Figure 6a). After five minutes of unrestricted communication, a host controller caps only traffic between islands A and B at 500 Mbit/s (or 100 Mbit/s) for five minutes, then removes the caps for five minutes of recovery. Each source interface applies a separate bandwidth cap to each affected peer based on its destination IP address. The cap applies independently in each direction, while all other worker pairs remain unrestricted. Transitions follow a common monotonic host clock independently of training progress. DDP uses TorchFT with NCCL sockets, while SPICE exchanges sparse payloads with four routing peers through a TCP transport that stages tensors in CPU memory. A calibration during the initial healthy phase sets the exchange deadline of the windowed version of SPICE to twice the pooled 99th percentile of exchange duration. Only complete payloads with matching attempt and schema identities received before the deadline contribute to the update, allowing communication with healthy peers to remain useful when capped transfers miss the deadline.

MethodPeers qqDensity δ\delta134M NCCL (no cap)10 Gbps5 Gbps2 Gbps1 Gbps551M NCCL (no cap)10 Gbps5 Gbps2 Gbps1 Gbps
DDP——2:054:495:145:357:0823:2448:4051:5655:1669:38
GASLoC1—1:382:242:272:324:0321:4934:2834:5435:5457:04
GASLoC2—1:473:574:184:336:0124:5158:2263:1367:2694:20
SPICE23.125%1:341:381:401:401:4319:5621:1121:2121:3022:19
SPICE43.125%1:351:461:481:501:5520:1223:1723:3924:1725:56
SPICE26.25%1:341:441:461:461:5320:1522:4823:1223:2625:09
SPICE46.25%1:361:592:042:072:1720:2526:4627:5429:1332:20

Table 6. End-to-end training times for the 134M and 551M models. Times are reported as hours:minutes and rounded to the nearest minute. Each model group shows the uncapped NCCL setting and bandwidth caps. The lowest time per model size and bandwidth is highlighted in bold.

Random seeds and reporting precision

We evaluate each training configuration using a single random seed. Recent pretraining studies likewise report single-seed results at selected model scales, citing the computational cost of repeated runs [26, 11]. Given this cost, we prioritize allocating our computational budget to hyperparameter tuning rather than repeated training across seeds. Related distributed pretraining studies also report their main comparisons without across-seed uncertainty estimates [8, 17, 15].

As a closely related reference, GASLoC evaluates 134M-parameter models with 16 replicas over four random seeds, reporting validation-loss standard deviations of at most 0.011 across the evaluated methods and configurations [6]. Following its main-result reporting convention, we report validation losses rounded to two decimal places, evaluated on a held-out set of 4M tokens for the 134M and 551M models.

Hyperparameters

We select hyperparameters by grid search using validation loss on a second separate evaluation set of 1M tokens. For the 134M model, we search the peak inner learning rate over {1.4,2,2.8,4,5,6}×10−3\{1.4, 2, 2.8, 4, 5, 6\} \times10^{-3}, the outer learning rate over {0.8,0.9,1.0}\{0.8, 0.9, 1.0\}, and the outer Nesterov momentum over {0.1,0.2,0.3}\{0.1, 0.2, 0.3\}. For the 551M model, the corresponding learning rate grids are {3,4,5,6}×10−3\{3, 4, 5, 6\} \times10^{-3} and {0.8,0.9}\{0.8, 0.9\}, with the same outer momentum grid. The inner learning rate warms up from one thousandth of its peak over the first 10% of training steps, followed by cosine decay to 10% of the peak. The inner optimizer is AdamW with β1=0.9\beta_1 = 0.9. Following Belilovsky et al. [3], we set β2\beta_2 to 0.95, 0.975, 0.987, 0.994, and 0.997 for R=8,16,32,64,128R = 8, 16, 32, 64, 128, respectively. For OLMoE, after short preliminary runs to assess initial training stability, we fix the peak inner learning rate to 2×10−32 \times10^{-3}, with warmup over the first 5% of training steps and the same initial and final learning rate fractions. We use β1=β2=0.9\beta_1 = \beta_2 = 0.9, an outer learning rate of 0.9, and outer Nesterov momentum of 0.2.

Across all models, AdamW uses ϵ=10−8\epsilon= 10^{-8} and weight decay of 0.1. Inner gradients are clipped to a norm of 1.0, while the outer optimizer uses no gradient clipping and a constant learning rate. Unless specified otherwise, the Top-kk block size is fixed to b=4096b = 4096, and the error feedback decay factor is fixed to 0.95 following Sarfi et al. [38]. Table 7 reports the selected hyperparameters.

ModelPeers qqHPR=8R = 8R=16R = 16R=32R = 32R=64R = 64R=128R = 128
134M2ηin\eta_{in}2e-32e-32e-32e-32e-3
134M2ηout\eta_{out}0.90.90.90.80.9
134M2μout\mu_{out}0.20.30.30.30.3
134M4ηin\eta_{in}4e-32.8e-32e-32e-32e-3
134M4ηout\eta_{out}0.90.90.90.90.9
134M4μout\mu_{out}0.20.30.30.30.3
134M8ηin\eta_{in}—2.8e-32e-32e-32e-3
134M8ηout\eta_{out}—0.90.80.90.9
134M8μout\mu_{out}—0.30.30.30.3
134M16ηin\eta_{in}——2e-32.8e-32.8e-3
134M16ηout\eta_{out}——0.90.90.9
134M16μout\mu_{out}——0.30.30.2
134M32ηin\eta_{in}———4e-34e-3
134M32ηout\eta_{out}———0.90.8
134M32μout\mu_{out}———0.20.3
134M64ηin\eta_{in}————5e-3
134M64ηout\eta_{out}————0.9
134M64μout\mu_{out}————0.2
551M2ηin\eta_{in}4e-34e-33e-3——
551M2ηout\eta_{out}0.90.90.9——
551M2μout\mu_{out}0.30.30.3——
551M4ηin\eta_{in}5e-35e-34e-3——
551M4ηout\eta_{out}0.90.90.9——
551M4μout\mu_{out}0.10.20.2——
551M8ηin\eta_{in}—6e-35e-3——
551M8ηout\eta_{out}—0.90.9——
551M8μout\mu_{out}—0.20.2——
551M16ηin\eta_{in}——5e-3——
551M16ηout\eta_{out}——0.9——
551M16μout\mu_{out}——0.1——

Table 7. Selected hyperparameters for SPICE on the 134M and 551M models across replica counts.

References

  1. [1]Assran, M., Loizou, N., Ballas, N., and Rabbat, M. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning, pp. 344–353. PMLR, 2019.arxiv.org/abs/1811.10792
  2. [2]Basu, D., Data, D., Karakus, C., and Diggavi, S. Qsparse-local-sgd: distributed sgd with quantization, sparsification, and local computations. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc.arxiv.org/abs/1906.02367
  3. [3]Belilovsky, E., Horoi, S., Oyallon, E., and Thérien, B. How inner optimizers shape replica scaling, 2026. URL https://arxiv.org/abs/2610.19064.
  4. [4]Bisk, Y., Zellers, R., Bras, R. L., Gao, J.-F., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2019. URL https://api.semanticscholar.org/CorpusID:208290939.arxiv.org/abs/1911.11641
  5. [5]Boyd, S., Ghosh, A., Prabhakar, B., and Shah, D. Randomized gossip algorithms. IEEE transactions on information theory, 52(6):2508–2530, 2006.DOI
  6. [6]Cagnasso, P., Belilovsky, E., and Oyallon, E. Unifying local communications and local updates for llm pretraining, 2026. URL https://arxiv.org/abs/2606.11081.
  7. [7]DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Wang, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Wang, Q., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang, R., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang, T., Yun, T., Pei, T., Sun, T., Xiao, W. L., Zeng, W., Zhao, W., An, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Zhang, X., Chen, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Song, X., Shan, X., Zhou, X., Yang, X., Li, X., Su, X., Lin, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhu, Y. X., Zhang, Y. Xu, Y., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Yu, Y., Zheng, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Tang, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Wu, Y., Ou, Y., Zhu, Y., Wang, Y., Gong, Y., Zhou, Y., He, Y., Zha, Y., Xiong, Y., Ma, Y., Yan, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Wu, Z. F., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Huang, Z., Zhang, Z., Xie, Z., Zhang, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Xu, Z., Wu, Z., Zhang, Z., Li, Z., Guo, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Gao, Z., and Pan, Z. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437.
  8. [8]Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., and Shen, J. Diloco: Distributed low-communication training of language models. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024. URL https://openreview.net/forum?id=pICSfWkJIk.arxiv.org/abs/2311.08105
  9. [9]Fatkhullin, I., Tyurin, A., and Richtárik, P. Momentum provably improves error feedback! In Advances in Neural Information Processing Systems, volume 36, 2023.arxiv.org/abs/2305.15155
  10. [10]Fournier, L., Nabli, A., Aminbeidokhti, M., Pedersoli, M., Belilovsky, E., and Oyallon, E. WASH: Train your ensemble with communication-efficient weight shuffling, then average. In OPT 2024: Optimization for Machine Learning, 2024. URL https://openreview.net/forum?id=13gJIQB8oc.arxiv.org/abs/2405.17517
  11. [11]Gladstone, A., Nanduru, G., Islam, M. M., Han, P., Ha, H., Chadha, A., Du, Y., Ji, H., Li, J., and Iqbal, T. Energy-based transformers are scalable learners and thinkers. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=ZBj3Qp1bYg. B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumanni, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgelen, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj, I., Molboyog, I., Tufanov, I., Leonidatis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
  12. [12]Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J., and Sifre, L. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2203.15556
  13. [13]Horoi, S., Thérien, B., Wolf, G., and Belilovsky, E. Can model merging improve aggregation in diloco?, 2026. URL https://arxiv.org/abs/2607.03011.
  14. [14]Jaghouar, S., Ong, J. M., Basra, M., Obeid, F., Straube, J., Keiblinger, M., Bakouch, E., Atkins, L., Panahi, M., Goddard, C., Ryabinin, M., and Hagemann, J. Intellect-1 technical report, 2024. URL https://arxiv.org/abs/2412.01152.
  15. [15]Janson, P., Oyallon, E., and Belilovsky, E. Stabilizing native low-rank LLM pretraining. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=kb5Wjkqu8q.arxiv.org/abs/2602.12429
  16. [16]Karimireddy, S. P., Rebjock, Q., Stich, S., and Jaggi, M. Error feedback fixes SignSGD and other gradient compression schemes. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3252–3261. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/karimireddy19a.html.arxiv.org/abs/1901.09847
  17. [17]Kolehmainen, J., Blagojev, N., Kara, S., Donaghy, J., Nies, C., and Ersoy, O. Noloco: No-all-reduce low communication training method for large models, 2026. URL https://arxiv.org/abs/2506.10911.
  18. [18]Koloskova, A., Stich, S., and Jaggi, M. Decentralized stochastic optimization and gossip algorithms with compressed communication. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3478–3487. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/koloskova19a.html.arxiv.org/abs/1902.00340
  19. [19]Koloskova*, A., Lin*, T., Stich, S. U., and Jaggi, M. Decentralized deep learning with arbitrary communication compression. In International Conference on Learning Representations, 2020.arxiv.org/abs/1907.09356
  20. [21]Kopyshevskiy, D., Thérien, B., Pinto, N., and Oyallon, E. LACCO: Local accumulation with computation–communication overlap, 2026. URL https://arxiv.org/abs/2611.17991.
  21. [23]Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., and Liu, J. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pp. 5336–5346. Curran Associates Inc., 2017.arxiv.org/abs/1705.09056
  22. [24]Liang, W., Liu, T., Wright, L., Constable, W., Gu, A., Huang, C.-C., Zhang, I., Feng, W., Huang, H., Wang, J., Purandare, S., Nadathur, G., and Idreos, S. Torchtitan: One-stop pytorch native solution for production ready LLM pretraining. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=SFN6Wm7YBI.arxiv.org/abs/2410.06511
  23. [25]Lidin, J., Sarfi, A., Miahi, E., Anthony, Q., Chauhan, S., Pappas, E., Thérien, B., Belilovsky, E., and Dare, S. Covenant-72b: Pre-training a 72b llm with trustless peers over-the-internet, 2026. URL https://arxiv.org/abs/2603.08163.
  24. [26]Loeschcke, S., Toftrup, M., Kastoryano, M. J., Belongie, S., and Snæbjarnarson, V. Loq†: Low-rank adapters for quantized pretraining. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024.arxiv.org/abs/2405.16528
  25. [27]McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. y. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Singh, A. and Zhu, J. (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp. 1273–1282. PMLR, 20–22 Apr 2017. URL https://proceedings.mlr.press/v54/mcmahan17a.html.arxiv.org/abs/1602.05629
  26. [28]Mok, R. K. P., Zou, H., Yang, R., Koch, T., Katz-Bassett, E., and Claffiy, K. C. Measuring the network performance of Google Cloud Platform. In Proceedings of the 21st ACM Internet Measurement Conference, pp. 54–61, 2021. doi: 10.1145/3487552.3487862. URL https://doi.org/10.1145/3487552.3487862.
  27. [29]Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Morrison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., Gu, Y., Arora, S., Bhagía, A., Schwenk, D., Wadden, D., Wettig, A., Hui, B., Dettmers, T., Kiela, D., Farhadi, A., Smith, N. A., Koh, P. W., Singh, A., and Hajishirzi, H. OLMoE: Open mixture-of-experts language models. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=xXTkbTBmqq.arxiv.org/abs/2409.02060
  28. [30]Nabli, A. and Oyallon, E. Dadao: Decoupled accelerated decentralized asynchronous optimization. In International Conference on Machine Learning, pp. 25604–25626. PMLR, 2023.arxiv.org/abs/2208.00779
  29. [31]Nabli, A. and Oyallon, E. Decentralized asynchronous optimization with dadao allows decoupling and acceleration. Journal of Machine Learning Research, 26(207):1–48, 2025.
  30. [32]Nabli, A., Belilovsky, E., and Oyallon, E. A²CiD²: Accelerating synchronous communication in decentralized deep learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 47451–47474. Curran Associates, Inc., 2023. doi: 10.48550/075280-2054. URL https://openreview.net/forum?id=YE04ArkeZb.
  31. [33]Nabli, A., Fournier, L., ERBACHER, P., Serrano, L., Belilovsky, E., and Oyallon, E. ACCO: Accumulate while you communicate, hiding communications in distributed LLM training. In OPT 2024: Optimization for Machine Learning, 2024. URL https://openreview.net/forum?id=VrKAGrUqGe.
  32. [34]Patarasuk, P. and Yuan, X. Bandwidth optimal all-reduce algorithms for clusters of workstations. J. Parallel Distrib. Comput., 69(2):117–124, 2009.
  33. [35]Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=n6SCkn2QaG.arxiv.org/abs/2406.17557
  34. [36]Peng, B., Chen, L., Su, B., Quesnelle, J., Kingma, D. P., and qiang liu. Demo: Decoupled momentum optimization. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=U9oewpa7cn.
  35. [37]Reddi, S. J., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=LKFG31B13U5.arxiv.org/abs/2003.00295
  36. [38]Sarfi, A., Thérien, B., Lidin, J., and Belilovsky, E. Overcoming the communication-performance tradeoff in llm pretraining, 2026. URL https://arxiv.org/abs/2508.15706.
  37. [39]Song, Z., Li, W., Jin, K., Shi, L., Yan, M., Yin, W., and Yuan, K. Communication-efficient topologies for decentralized learning with $o(1)$ consensus rate. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=AyiiHcRzTd.
  38. [40]Stich, S. U. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=S1g2JnRCFX.arxiv.org/abs/1805.09767
  39. [41]Stich, S. U. and Karimireddy, S. P. The error-feedback framework: Sgd with delayed gradients. Journal of Machine Learning Research, 21(37):1–36, 2020. URL http://jmlr.org/papers/v21/19-748.html.
  40. [42]Stich, S. U., Cordonnier, J.-B., and Jaggi, M. Sparsified sgd with memory. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/b440509a0106086a67bc2ea9df0a1dab-Paper.pdf.arxiv.org/abs/1809.07599
  41. [43]Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J. F., Ju, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Koulamn, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.arxiv.org/abs/2307.09288
  42. [44]Vogels, T., Hendrikx, H., and Jaggi, M. Beyond spectral gap: the role of the topology in decentralized learning. J. Mach. Learn. Res., 24(1), January 2023. ISSN 1532-4435.
  43. [45]Wang, J., Sahu, A. K., Yang, Z., Joshi, G., and Kar, S. Matcha: Speeding up decentralized sgd via matching decomposition sampling. In 2019 Sixth Indian Control Conference (ICC), pp. 299–300, 2019.arxiv.org/abs/1905.09435
  44. [46]Wang, J., Lu, Y., Yuan, B., Chen, B., Liang, P., De Sa, C., Re, C., and Zhang, C. CocktailSGD: Fine-tuning foundation models over 500Mbps networks. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 36058–36076. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/wang23t.html.
  45. [47]Wang, Z., Zhang, J., Wu, X., and Johansson, M. From promise to practice: Realizing high-performance decentralized training. In Yue, Y., Garg, A., Peng, N., Sha, F., and Yu, R. (eds.), International Conference on Learning Representations, volume 2025, pp. 26204–26249, 2025.arxiv.org/abs/2410.11998
  46. [48]Ying, B., Yuan, K., Chen, Y., Hu, H., PAN, P., and Yin, W. Exponential graph is provably efficient for decentralized deep training. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 13975–13987. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/74e1ed8b55ea44fd7dbb685c412568a4-Paper.pdf.arxiv.org/abs/2110.13363
  47. [49]Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Korhonen, A., Traüm, D., and Márquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472/.DOI

Paper details

vlnc:W_hg6sdk7p
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude, ChatGPT

View statement
Looking for related papers…
Contents