2026
Out of the Memory Barrier: A Highly Memory-Efficient Training System for LLMs with Million-Token Contexts
ICLR 2026poster
Training Large Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU memory overhead, not training time. The primary culprits are the activations, whose memory footprints scale linearly with sequence length. We introduce OOMB, a highly memory-efficient training system th…