Athanasios Glentis

Memory-Efficient LLM Pretraining via Minimalist Optimizer Design

Training large language models (LLMs) typically relies on adaptive optimizers such as Adam, which introduce extra operations and require significant more memory to maintain first- and second-order moments than SGD. While recent works such as GaLore, Fira and APOLLO have proposed state-compressed var…