2026
FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models
ICLR 2026poster
Low-rank optimization has emerged as a promising direction in training large language models (LLMs) to improve running time and reduce the memory usage of adaptive optimizers by constraining learning to a lower-dimensional space. Prior work typically projects gradients of linear layers using approac…