2025
Tensor-Parallelism with Partially Synchronized Activations
NeurIPS 2025poster
Training and inference of Large Language Models (LLMs) with tensor-parallelism requires substantial communication to synchronize activations. Our findings suggest that with a few minor adjustments to current practices, LLMs can be trained without fully synchronizing activations, reducing bandwidth d…