2026
The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
ICLR 2026poster
Recent efforts to accelerate LLM pretraining have focused on computationally-efficient approximations that exploit second-order structure. This raises a key question for large-scale training: how much performance is forfeited by these approximations? To probe this question, we establish a practical…