Fırat Öncel

Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?

In the last decade, the generalization and adaptation abilities of deep learning models were typically evaluated on fixed training and test distributions. Contrary to traditional deep learning, large language models (LLMs) are (i) even more overparameterized, (ii) trained on unlabeled text corpora c…