2025
First Attentions Last: Better Exploiting First Attentions for Efficient Parallel Training
NeurIPS 2025poster
As training billion-scale transformers becomes increasingly common, employing multiple distributed GPUs along with parallel training methods has become a standard practice. However, existing transformer designs suffer from significant communication overhead, especially in Tensor Parallelism (TP), wh…