Sun Ao

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training

Training large language models (LLMs) heavily relies on distributed training strategies, among which pipeline parallelism (PP) plays a crucial role. As training sequences extend to 32k or even 128k tokens, current PP methods face severe bottlenecks, including substantial pipeline bubbles and high me…