Hyukju Na

First Attentions Last: Better Exploiting First Attentions for Efficient Parallel Training

As training billion-scale transformers becomes increasingly common, employing multiple distributed GPUs along with parallel training methods has become a standard practice. However, existing transformer designs suffer from significant communication overhead, especially in Tensor Parallelism (TP), wh…