Hyoungjoo Kim

BPipe: Memory-Balanced Pipeline Parallelism for Training Large Language Models

Pipeline parallelism is a key technique for training large language models within GPU clusters. However, it often leads to a memory imbalance problem, where certain GPUs face high memory pressure while others underutilize their capacity. This imbalance results in suboptimal training performance, eve…