Amaury Sabran

Comparing the learning dynamics of in-context learning and fine-tuning in language models

Pretrained language models can acquire novel tasks either through in-context learning (ICL)---adapting behavior via activations without weight updates---or through supervised fine-tuning (SFT), where parameters are explicitly updated. Prior work has reported differences in their generalization perfo…