2025
To Code or Not To Code? Exploring Impact of Code in Pre-training
ICLR 2025poster
Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vital role in general LLMs' performance, there is only limited w…