In this paper
AbstractIntroductionBackground and PreliminariesModel-Parallel LLM TrainingLow-Precision Compression for LLM TrainingNVFP4 Representation and ScalingFP4Quant: Ultra-Low-Precision Communication FrameworkMicro-Grouped Dual-Scale FP4 QuantizationHybrid Two-Level FWHT PipelineWarp-Cooperative FP4 PackingEvaluationExperimental SetupConvergence and Model QualityDownstream Task AccuracySystem EfficiencyConclusion and Future WorkAcknowledgementsReferencesWorksComputer Science
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
Abstract
The rapid scaling of large language models (LLMs) has made communication overhead a critical bottleneck in distributed model-parallel training. Although existing low-precision compression methods can effectively reduce communication costs, further reducing the communication footprint to ultra-low-bit formats such as FP4 introduces new challenges, including limited numerical range, sub-byte storage overhead, and increased computational cost from fine-grained quantization. In this paper, we present FP4Quant, a micro-grouped FP4 compression framework for efficient model-parallel LLM training. FP4Quant integrates a micro-grouped dual-scale FP4 quantization mechanism to preserve local tensor distribution characteristics, a warp-cooperative FP4 packing scheme to enable efficient sub-byte data representation, and a hybrid two-level FWHT pipeline to mitigate quantization errors with GPU-aware execution optimization. Experiments on NVIDIA H100 GPUs with Qwen-7B demonstrate up to 1.56 end-to-end throughput improvement over BF16, revealing a trade-off between throughput and model quality.
Introduction
In recent years, large language models (LLMs) have advanced rapidly, with parameter scales growing exponentially to reach the trillion-parameter regime [5, 8, 12]. Training such models requires hundreds to thousands of GPUs, making model parallelism (MP) essential for efficiently distributing computational workloads across devices, typically through tensor parallelism (TP) and pipeline parallelism (PP) [20, 17]. As model sizes continue to increase, communication overhead, particularly AllReduce operations in TP, has increasingly become a critical performance bottleneck in distributed LLM training [22, 20, 23, 13].
To mitigate communication bottlenecks, a variety of optimization approaches have been proposed and actively explored in recent years. For DP, gradient compression techniques, including quantization [15, 3, 4, 27], sparsification [2, 26], and local accumulation [24], have been extensively studied to reduce communication costs. For TP, frameworks such as TACO [19] demonstrate that adaptive FP8-based compression can significantly reduce communication overhead while maintaining near-lossless training accuracy. More broadly, FP8-based methods (e.g., FP8-LM [21] and COAT [29]) improve training efficiency by extending low-precision representations to gradients, optimizer states, and activations.
However, the limited compression ratio of FP8 is often insufficient to fully alleviate communication bottlenecks at scale, motivating more aggressive low-bit approaches in distributed settings. While 4-bit quantization offers the potential for substantially higher compression, it introduces fundamental challenges beyond simple precision scaling [9, 10]. Recent efforts have explored FP4 techniques for model computation [6, 7, 1] and low-bit communication in distributed training [15, 11]. Yet, these methods primarily target computation, DP or PP, leaving TP communication under extreme low-bit constraints largely unexplored.
Despite its potential to reduce communication overhead, directly applying coarse-grained 4-bit quantization to model-parallel communication faces three key challenges. First, FP4’s narrow dynamic range causes long-tail values to overflow and near-zero values to round to zero, distorting gradient directions and destabilizing early-stage training [28, 19]. Second, sub-byte storage requires packing two 4-bit values per byte, while naive packing incurs additional data movement and bit-manipulation overhead, limiting communication efficiency [16]. Third, reducing 4-bit quantization error requires fine-grained quantization, but finer granularity increases scaling-factor computation, potentially making quantization itself a system bottleneck [30]. To address these challenges, we propose FP4Quant, a micro-grouped FP4 compression framework that jointly optimizes representation fidelity and GPU execution efficiency for scalable LLM distributed training. The key contributions are summarized as follows:
We propose a micro-grouped dual-scale FP4 quantization mechanism to preserve local tensor distributions under ultra-low-bit compression. Micro-grouping captures local variations, while dual-scale adaptive scaling and max-absolute scaling better match the FP4 dynamic range and reduce overflow.
We introduce a hybrid two-level FWHT pipeline to support fine-grained FP4 compression with limited computational overhead. Intra-warp butterfly operations are efficiently implemented using register-level shuffles, while cross-warp transformations are executed using shared memory, reducing transformation overhead and enabling efficient execution on modern GPUs.
We further develop a warp-cooperative FP4 packing scheme for efficient sub-byte FP4 storage. The kernel exploits warp-level primitives to exchange quantized values and performs nibble-level packing directly in registers, reducing intermediate memory operations and enabling aligned byte-wise storage.
Experiments on NVIDIA H100 clusters show that FP4Quant improves end-to-end training throughput by up to compared to BF16, while main-
Vector (PDF)
caption: Example distributed training configuration with model parallelism, with 4 tensor-parallel groups, and 2 pipeline stages.
taining comparable convergence performance on Qwen models. FP4Quant provides an efficient and scalable solution for ultra-low-bit communication in model-parallel LLM training under practical TP and PP configurations.
Background and Preliminaries
Model-Parallel LLM Training
Model parallelism, including TP and PP, is widely adopted in large-scale LLM training to distribute computation and model parameters across GPUs. As shown in Fig. 1, TP partitions individual layers across devices and relies on frequent collective communication operations (e.g., AllReduce) to exchange intermediate tensors [23], leading to substantial overhead as model size and the degree of parallelism increase [18]. In contrast, PP divides the model into stages and transfers activations and gradients between them, where communication efficiency directly affects pipeline utilization and overall throughput [14]. To alleviate these costs, recent studies have explored low-precision compression techniques to reduce communication volume while preserving training accuracy [21, 19]. However, extending these techniques to ultra-low-bit formats (e.g., FP4) remains challenging, as it is difficult to balance high compression ratios with stable training convergence.
Low-Precision Compression for LLM Training
Recent studies have explored 4-bit floating-point quantization to improve the efficiency of large-scale LLM training. METIS [6] reduces quantization error via spectral-domain FP4 design for accurate W4A4G4 training, while Quartet [7] enables native FP4 training through hardware-aware optimization. NVIDIA’s NVFP4 pre-training work [1] demonstrates scalable FP4 training with optimized scaling and mixed-precision scheduling. Subsequent efforts further enhance stability by addressing FP4’s limited dynamic range, including differentiable quantization and outlier-aware scaling [25], as well as adaptive mixed-precision strategies across model components and training stages [31].
Beyond computation-oriented FP4 training, several studies have explored low-bit communication compression techniques for distributed LLM training. SDP4Bit [15] investigates 4-bit gradient communication in sharded data-parallel training and shows that substantial communication reduction can be achieved with limited accuracy degradation. Tahquant [11] explores low-bit activation compression for pipeline-parallel training to reduce inter-stage communication overhead. However, existing low-precision techniques primarily focus on computation acceleration, DP or PP communication, whereas the application of ultra-low-bit quantization to TP communication remains relatively underexplored.
NVFP4 Representation and Scaling
NVFP4 is a 4-bit floating-point format designed for efficient AI workloads, retaining sign, exponent, and mantissa components. To compensate for its limited dynamic range, NVFP4 employs block-level scaling, in which multiple FP4 elements share a higher-precision scaling factor. Given an input tensor , NVFP4 represents it as , where denotes the quantized FP4 values and is the corresponding scaling factor. NVFP4 defines two formats, E2M1 and E3M0 [1], both following the representation . E2M1 uses one sign bit, two exponent bits, and one mantissa bit, whereas E3M0 allocates one sign bit and three exponent bits without a mantissa. Despite this design, the extremely limited representation results in rapidly increasing quantization step sizes, leading to larger errors compared with higher-precision formats.
Although block-level scaling improves accuracy over naive FP4 quantization, fixed-size blocks remain insufficient for capturing fine-grained variations in LLM tensors. Large blocks are more sensitive to outliers, whereas smaller blocks incur higher scaling overhead. This trade-off motivates the development of adaptive scaling strategies and efficient GPU implementations to enable accurate and practical NVFP4-based communication.
FP4Quant: Ultra-Low-Precision Communication Framework
Motivated by the challenges of ultra-low-bit communication, we propose FP4Quant, a micro-grouped FP4 compression framework integrated into the collective communication path. As illustrated in Figure 2, the compression workflow consists of three main components. Section 3.1 describes the micro-grouped dual-scale FP4 quantization mechanism (①); Section 3.2 presents the hybrid two-level FWHT pipeline (②); and Section 3.3 introduces warp-cooperative FP4 packing (③). At the receiver, the compressed payload is reconstructed through unpacking, de-quantization, and the inverse FWHT. Together, these components address the numerical and execution challenges of FP4-based communication compression for model-parallel LLM training.
Figure 2. Overview of FP4Quant.
Micro-Grouped Dual-Scale FP4 Quantization
FP4Quant adopts a micro-grouped dual-scale scheme to address the limited dynamic range of NVFP4, combining fine-grained grouping with two-stage scaling for accurate distribution adaptation and efficient FP4 encoding.
Micro-grouping. Each communication tensor is partitioned into contiguous micro-groups of elements, where . Each group is processed independently to capture localized statistics, enabling fine-grained and adaptive scaling while preserving high-efficiency GPU execution.
Dual-scale quantization. For each group , FP4Quant first computes the second moment and derives an adaptive scale
where avoids numerical instability. By normalizing the RMS magnitude of each group, maps diverse local distributions into the effective FP4 range. After the FWHT transformation in § 3.2, the transformed values are quantized using a group-wise max-absolute scale:
Here, performs pre-rotation normalization, while determines the final quantization range. Both scales are stored per group and used for reconstruction.
Hybrid Two-Level FWHT Pipeline
Motivation. FP4 quantization is sensitive to outliers, where a few extreme values can dominate the max-absolute scale and reduce effective precision. FP4Quant applies an orthonormal Walsh–Hadamard transform (FWHT) to redistribute energy across dimensions, producing a more balanced representation (see Figure 3). The transform is self-inverse, allowing the same operations for decompression.
Figure 3. Overview of the hybrid two-level FWHT pipeline.
Two-level execution. A length- FWHT consists of butterfly stages. According to the GPU execution hierarchy, FP4Quant divides these operations into two levels. For intra-warp stages with butterfly distances , paired elements are exchanged through register shuffle operations without shared-memory level:
which corresponds to the butterfly operation shown in Figure 4. These stages are fully unrolled and executed in registers. For cross-warp stages with , intermediate results are exchanged through shared memory with synchronization barriers. Only such stages are required, while groups with complete the entire transform using register-level operations. The final output is normalized by to obtain the orthonormal transform.. This design aligns FWHT stages with GPU communication primitives, minimizing synchronization overhead while effectively smoothing outliers for FP4 quantization.
Figure 4. Illustration of the butterfly operation in FWHT.
Warp-Cooperative FP4 Packing
FP4 codes occupy four bits, requiring two adjacent codes to be stored in one byte. A straightforward implementation may introduce intermediate memory accesses for byte assembly, where additional loads, shifts, and masks increase memory traffic and reduce the benefit of FP4 compression. FP4Quant therefore performs nibble packing directly in registers using warp-level shuffle operations.
After quantization, thread holds one 4-bit code in its low nibble. Adjacent threads exchange their codes through a stride-1 butterfly shuffle:
For even lanes (), the shuffled value provides the high nibble from the adjacent thread, while the local code remains as the low nibble. Each even lane then assembles one byte as and performs a single coalesced store, whereas odd lanes remain inactive. This design avoids shared-memory usage and additional memory round trips, producing naturally aligned byte-level writes across the warp. During decompression, the process is reversed: each thread loads the corresponding packed byte and extracts either the low or high nibble according to the parity of its index before dequantization. This warp-cooperative design eliminates intermediate memory traffic and shared-memory usage, achieving near-zero packing overhead with efficient, aligned writes.
Evaluation
Experimental Setup
Platform. All experiments are conducted on a single node equipped with eight NVIDIA H100 GPUs (80 GB HBM3 each; driver 595.71.05; CUDA 13.2), interconnected via NVLink. FP4Quant is implemented as an NCCL compressor plugin and evaluated within a production-grade model-parallel training stack.
Model, data, and task. We train Qwen-7B on the OpenWebMath corpus. Model quality is evaluated using validation loss and the corresponding perplexity (PPL) (PPL = ). System efficiency is measured by end-to-end training throughput (TFLOP/s per GPU) under different parallel settings. Downstream model capability is further evaluated using accuracy on CMMLU and C-Eval.
Parallel configurations. We evaluate FP4Quant under three parallel regimes: TP, PP, and their combined MP configuration in large-scale distributed training. For each regime, we investigate micro-group sizes and compare against two baselines: the uncompressed BF16 (Ring) and Tahquant, a representative method for activation communication compression in PP settings.
Convergence and Model Quality
To evaluate the convergence behavior and model quality of FP4Quant under ultra-low-bit communication compression, we measure the validation loss and perplexity across different micro-group sizes (). Table 2 summarizes the final validation metrics, while Figure 5 presents the convergence curves over 10, 000 iterations under TP and PP configurations.
Figure 5. Validation loss of Qwen-7B with different micro-group sizes under TP and PP communication compression.
| Method | TP | PP | ||||||||
| Loss | PPL | TFLOP/s | Speedup | Loss | PPL | TFLOP/s | Speedup | |||
| BF16 (Ring) | 2.1505 | 8.5893 | 111.0 | 2.1505 | 8.5893 | 189.1 | ||||
| Tahquant | 3.4848 | 32.6163 | 197.4 | 2.1653 | 8.7174 | 189.6 | ||||
| FP4Quant() | 2.2306 | 9.3050 | 155.3 | 2.1525 | 8.6105 | 190.1 | ||||
| FP4Quant() | 2.4223 | 11.2713 | 171.2 | 2.1552 | 8.6298 | 191.1 | ||||
| FP4Quant() | 3.0616 | 21.3613 | 173.3 | 2.1626 | 8.6940 | 190.7 | ||||
| FP4Quant() | 3.3512 | 28.5368 | 172.6 | 3.4270 | 30.7830 | 190.3 |
Table 2. Comparison of training accuracy and throughput between FP4Quant and baselines (BF16 and Tahquant) under TP4 and PP on Qwen-7B.
The results demonstrate that FP4Quant consistently outperforms Tahquant in preserving model quality under FP4 communication compression. Under TP, Tahquant suffers from severe accuracy degradation (loss 3.4848, PPL = 32.6163), whereas FP4Quant with achieves substantially lower loss (2.2306, PPL = 9.3050) and maintains a stable convergence trajectory close to the full-precision baseline, as illustrated in Figure 5. Under PP, FP4Quant further demonstrates near-lossless performance (loss 2.1525, PPL = 8.6105), matching BF16 accuracy and outperforming Tahquant.
In addition, FP4Quant exhibits clear sensitivity to group size, where increasing leads to gradual degradation in TP and a sharp breakdown at large (e.g., ) in both TP and PP. This trend suggests that overly coarse grouping introduces scale mismatch and amplifies quantization error. Nevertheless, with appropriately chosen group sizes (e.g., ), FP4Quant achieves an excellent balance between accuracy and communication efficiency and consistently delivers superior performance compared to Tahquant.
Downstream Task Accuracy
Training loss and downstream accuracy reflect different aspects of model performance: loss measures optimization convergence, whereas downstream tasks evaluate practical model capability. Although the two metrics are not strictly correlated, poor convergence or substantially higher loss than the baseline typically leads to degraded downstream performance. Table 1 reports evaluation results on CMMLU and C-Eval (valid) with . FP4Quant consistently preserves downstream performance across model-parallel settings, achieving scores of 0.5287/0.4829 (TP), 0.5288/0.5342 (PP), and 0.5301/0.5253 (MP), compared with the BF16 baseline (0.5365/0.5215). Notably, under PP and MP configurations, FP4Quant matches or slightly exceeds BF16 on C-Eval, demonstrating strong capability preservation under ultra-low-bit compression.
| Configuration | Method | CMMLU | C-Eval (valid) |
| BF16 (Ring) | 0.5365 0.0045 | 0.5215 0.0133 | |
| TP | FP4Quant | 0.5287 0.0045 | 0.4829 0.0133 |
| TP | Tahquant | 0.2530 0.0040 | 0.2771 0.0122 |
| PP | FP4Quant | 0.5288 0.0045 | 0.5342 0.0133 |
| PP | Tahquant | 0.5179 0.0045 | 0.5186 0.0133 |
| MP | FP4Quant | 0.5301 0.0045 | 0.5253 0.0133 |
| MP | Tahquant | 0.2534 0.0040 | 0.2422 0.0117 |
Table 1. Downstream accuracy on CMMLU and C-Eval (valid) with under different model-parallel configurations.
In contrast, TahQuant exhibits severe degradation under communication-intensive settings. Under TP and MP, its performance drops to near random-chance levels (0.2530/0.2771 and 0.2534/0.2422), while remaining stable only under PP. This suggests that coarse-grained 4-bit quantization is insufficient to maintain optimization quality, ultimately degrading downstream performance. Overall, FP4Quant achieves both consistently stable convergence and superior downstream fidelity compared with existing 4-bit communication methods.
System Efficiency
Table 2 shows that FP4Quant achieves a superior balance between training accuracy and throughput compared to BF16. Under TP, Tahquant attains the highest throughput (197.4 TFLOP/s/GPU, ), but suffers from severe accuracy degradation, indicating that its aggressive quantization compromises training fidelity. In contrast, FP4Quant substantially improves model quality while still providing significant speedup. At , FP4Quant reduces the loss to 2.2306 while achieving a speedup over BF16, and continues to offer a favorable trade-off at larger group sizes. For example, at , FP4Quant reaches 173.3 TFLOP/s/GPU while achieving significantly better accuracy than Tahquant. Although FP4Quant achieves lower throughput than Tahquant, it avoids severe accuracy degradation while delivering a better accuracy–efficiency trade-off.
Under PP, where computation dominates execution and communication overhead is less significant, all methods achieve comparable throughput (189–191 TFLOP/s/GPU). FP4Quant reaches 190.1–191.1 TFLOP/s/GPU, closely matching the BF16 baseline (189.1 TFLOP/s/GPU) with up to speedup, indicating that its quantization and scaling overhead is effectively hidden by computation. Unlike communication-bound TP training, PP offers limited acceleration potential because performance is primarily determined by computation throughput. Overall, FP4Quant enables efficient low-bit communication with substantial acceleration under TP while incurring negligible runtime overhead under PP, demonstrating its effectiveness across diverse parallel configurations.
Conclusion and Future Work
In this paper, we present FP4Quant, an efficient FP4-based communication compression framework for model-parallel LLM training. It integrates micro-grouped dual-scale quantization, warp-cooperative packing, and a hybrid FWHT-based pipeline to improve quantization fidelity and system efficiency. Experiments show that FP4Quant achieves up to end-to-end throughput improvement while preserving convergence. Our evaluation focuses on Qwen-7B on a single-node eight-GPU H100 system. Future work will investigate scalability across model sizes and multi-node environments, adaptive precision strategies, and applications to heterogeneous systems, multimodal models, and reinforcement learning.
Acknowledgements
This work was supported by the National Natural Science Foundation of China through the Excellent Young Scientists Fund Program (Overseas) (Project No. E401470), the National Key Research and Development Program of China (Grant Nos. 2020YFA0907000 and 2025YFB30037002), the National Natural Science Foundation of China (Grant Nos. 62032023 and T2125013), and Ant Group. The AI-driven experiments, simulations, and model training were performed on the robotic AI-Scientist platform of the Chinese Academy of Sciences.
References
- [1]Abecassis, F., Agrusa, A., Ahn, D., Alben, J., Alborghetti, S., Andersch, M., Arayandi, S., Bjorlin, A., Blakeman, A., Briones, E., et al.: Pretraining large language models with nvfp4. arXiv preprint arXiv:2509.25149 (2025)arxiv.org/abs/2509.25149
- [2]Aji, A.F., Heafield, K.: Sparse communication for distributed gradient descent. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. p. 440–445. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/d17-1045, http://dx.doi.org/10.18653/v1/D17-1045
- [3]Alistarh, D., Grubic, D., Li, J.Z., Tomioka, R., Vojnovic, M.: Qsgd: communication-efficient sgd via gradient quantization and encoding. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 1707–1718. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)
- [4]Bernstein, J., Wang, Y.X., Azizzadenesheli, K., Anandakumar, A.: signsgd: Compressed optimisation for non-convex problems (2018), https://arxiv.org/abs/1802.04434
- [5]Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners (gpt-3). arXiv preprint arXiv:2005.14165 75 (2020)arxiv.org/abs/2005.14165
- [6]Cao, H., Chen, M., Yang, Y., Huang, R., Dong, F., Zhou, J., Chen, A., Dong, M., Wang, Y., Hou, J., Cheng, Y., Wu, F., Yang, F., Lu, T., Gu, N., Shang, L.: Metis: Training llms with fp4 quantization (2025), https://arxiv.org/abs/2509.00404
- [7]Castro, R., Panferov, A., Tabesh, R., Sieberling, O., Chen, J., Nikdan, M., Ashkboos, S., Alistarh, D.: Quartet: Native fp4 training can be optimal for large language models. Advances in Neural Information Processing Systems 38, 43552–43572 (2026)arxiv.org/abs/2505.14669
- [8]Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)arxiv.org/abs/2204.02311
- [9]Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L.: Llm.int8(): 8-bit matrix multiplication for transformers at scale. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22, Curran Associates Inc., Red Hook, NY, USA (2022)arxiv.org/abs/2208.07339
- [10]Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D.: Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)arxiv.org/abs/2210.17323
- [11]He, G., Cao, Y., He, Y., Bai, T., Chen, K., Yuan, K., Yuan, B.: Tah-quant: Effective activation quantization in pipeline parallelism over slow network (2026), https://arxiv.org/abs/2506.01352
- [12]Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L.A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J.W., Sifre, L.: Training compute-optimal large language models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22, Curran Associates Inc., Red Hook, NY, USA (2022)arxiv.org/abs/2203.15556
- [13]Hosoki, R., Sato, K., Endo, T., Bigot, J., Audit, E.: Optimizing intra-layer parallel communication for llm training on systems with fully-connected mesh gpu topology. In: Proceedings of the Supercomputing Asia and International Conference on High Performance Computing in Asia Pacific Region. p. 328–339. SCA/HPCAsia ’26, Association for Computing Machinery, New York, NY, USA (2026). https://doi.org/10.1145/3773656.3773675, https://doi.org/10.1145/3773656.3773675 M. Liu et al.
- [14]Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M.X., Chen, D., Lee, H., Ngiam, J., Lee, Q.V., Wu, Y., Chen, Z.: Gpipe: Efficient training of giant neural networks using pipeline parallelism (2019), https://arxiv.org/abs/1811.06965
- [15]Jia, J., Xie, C., Lu, H., Wang, D., Feng, H., Zhang, C., Sun, B., Lin, H., Zhang, Z., Liu, X., Tao, D.: Sdp4bit: toward 4-bit communication quantization in sharded data parallelism for llm training. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. NIPS ’24, Curran Associates Inc., Red Hook, NY, USA (2024)arxiv.org/abs/2410.15526
- [16]Kamath, A.K., Krishnamurthy, A., Canini, M., Peter, S.: Reducing the gpu memory bottleneck with lossless compression for ml. In: Proceedings of the 21st European Conference on Computer Systems. p. 899–918. EUROSYS ’26, Association for Computing Machinery, New York, NY, USA (2026). https://doi.org/10.1145/3767295.3803595, https://doi.org/10.1145/3767295.3803595. 3803595
- [17]Korthikanti, V., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., Catanzaro, B.: Reducing activation recomputation in large transformer models (2022), https://arxiv.org/abs/2205.05198
- [18]Korthikanti, V., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., Catanzaro, B.: Reducing activation recomputation in large transformer models (2022), https://arxiv.org/abs/2205.05198
- [19]Liu, M., Liu, X., Tian, X., Lu, B., Lyu, S., Yin, S., Huang, W., Wei, Z., Zhao, H., Tan, G., Tao, D.: Taco: Efficient communication compression of intermediate tensors for scalable tensor-parallel llm training. In: Proceedings of the 35th International Symposium on High-Performance Parallel and Distributed Computing. p. 235–248. HPDC ’26, Association for Computing Machinery, New York, NY, USA (2026). https://doi.org/10.1145/3806645.3807584, https://doi.org/10.1145/3806645.3807584
- [20]Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., Zaharia, M.: Efficient large-scale language model training on gpu clusters using megatron-lm. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3458817.3476209, https://doi.org/10.1145/3458817.3476209
- [21]Peng, H., Wu, K., Wei, Y., Zhao, G., Yang, Y., Liu, Z., Xiong, Y., Yang, Z., Ni, B., Hu, J., Li, R., Zhang, M., Li, C., Ning, J., Wang, R., Zhang, Z., Liu, S., Chau, J., Hu, H., Cheng, P.: Fp8-lm: Training fp8 large language models (2023), https://arxiv.org/abs/2310.18313
- [22]Rajbhandari, S., Rasley, J., Ruwase, O., He, Y.: Zero: memory optimizations toward training trillion parameter models. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’20, IEEE Press (2020)arxiv.org/abs/1910.02054
- [23]Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., Catanzaro, B.: Megatron-lm: Training multi-billion parameter language models using model parallelism (2020), https://arxiv.org/abs/1909.08053
- [24]Stich, S.U.: Local sgd converges fast and communicates little (2019), https://arxiv.org/abs/1805.09767
- [25]Wang, R., Gong, Y., Liu, X., Zhao, G., Yang, Z., Guo, B., Zha, Z., Cheng, P.: Optimizing large language model training using fp4 quantization (2025), https://arxiv.org/abs/2501.17116
- [26]Wangni, J., Wang, J., Liu, J., Zhang, T.: Gradient sparsification for communication-efficient distributed optimization. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. p. 1306–1316. NIPS’18, Curran Associates Inc., Red Hook, NY, USA (2018)
- [27]Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., Li, H.: Terngrad: ternary gradients to reduce communication in distributed deep learning. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 1508–1518. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)arxiv.org/abs/1705.07878
- [28]Wu, X., Li, C., Aminabadi, R.Y., Yao, Z., He, Y.: Understanding int4 quantization for language models: latency speedup, composability, and failure cases. In: Proceedings of the 40th International Conference on Machine Learning. ICML’23, JMLR.org (2023)arxiv.org/abs/2301.12017
- [29]Xi, H., Cai, H., Zhu, L., Lu, Y., Keutzer, K., Chen, J., Han, S.: Coat: Compressing optimizer states and activations for memory-efficient fp8 training. In: International Conference on Learning Representations. vol. 2025, pp. 42989–43011 (2025)arxiv.org/abs/2410.19313
- [30]Zhang, L., Fei, W., Wu, W., He, Y., Lou, Z., Zhou, H.: Dual grained quantization: Efficient fine-grained quantization for llm (2023), https://arxiv.org/abs/2310.04836
- [31]Zhou, J., Tang, D., Fu, R., Hu, B., Xu, H., Wang, Y., Pei, Z., Su, Z., Liu, L., Zhang, X., Zhang, W.: Towards efficient pre-training: Exploring fp4 precision in large language models (2025), https://arxiv.org/abs/2502.11458



