Zhouyang Liu

Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio

The impressive performance of large language models (LLMs) arises from their massive scale and heterogeneous module composition. However, this structural heterogeneity poses significant optimization challenges. While adaptive optimizers such as Adam(W) provide per-parameter adaptivity, they do not e