Chan Wu

Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models

Recently, various strategies for distributed training of large language models (LLMs) have been proposed. By categorizing them into basic strategies and composite strategies, we have discovered that existing basic strategies provide limited options in specific scenarios, leaving considerable room fo…