3 accepted papers
Low-precision formats such as BF16 and FP8 can greatly improve training efficiency, but fully low-precision training often degrades accuracy under standard optimizers. We identify a key cause: additive updates can vanish under coarse mantissa resolution. We introduce M+Adam, an optimizer for stable …
Despite advances using low-rank adapters and quantization, pretraining of large models on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose Low-Rank Adapters for Quantized Training (LoQT…
Given a set of points, clustering consists of finding a partition of a point set into $k$ clusters such that the center to which a point is assigned is as close as possible. Most commonly, centers are points themselves, which leads to the famous $k$-median and $k$-means objectives. One may also choo…