2024
Collage: Light-Weight Low-Precision Strategy for LLM Training
ICML 2024poster
Large models training is plagued by the intense compute cost and limited hardware memory. A practical solution is low-precision representation but is troubled by loss in numerical accuracy and unstable training rendering the model less useful. We argue that low-precision floating points can perform…