2025
Hierachical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM
NeurIPS 2025poster
Training Long-Context Large Language Models (LLMs) is challenging, as hybrid training with long-context and short-context data often leads to workload imbalances. Existing works mainly use data packing to alleviate this issue, but fail to consider imbalanced attention computation and wasted communic…