Pei-Hsuan Chang

Outlier-Efficient Hopfield Layers for Large Transformer-Based Models

We introduce an Outlier-Efficient Modern Hopfield Model (termed `OutEffHop`) and use it to address the outlier inefficiency problem of training gigantic transformer-based models. Our main contribution is a novel associative memory model facilitating _outlier-efficient_ associative memory retrievals.…