Chung-Ching Chang

Memory Augmented Language Models through Mixture of Word Experts

Scaling up the number of parameters of language models has proven to be an effective approach to improve performance. For dense models, increasing their size proportionally increases their computational footprint. In this work, we seek to aggressively decouple learning capacity and FLOPs through Mix…