Sebastian Loeschcke

M+Adam: Low-Precision Training via Mantissa–Exponent Optimization

Low-precision formats such as BF16 and FP8 can greatly improve training efficiency, but fully low-precision training often degrades accuracy under standard optimizers. We identify a key cause: additive updates can vanish under coarse mantissa resolution. We introduce M+Adam, an optimizer for stable …