2024
Metric Transforms and Low Rank Representations of Kernels for Fast Attention
NeurIPS 2024spotlight
We introduce a new linear-algebraic tool based on group representation theory, and use it to address three key problems in machine learning. 1. Past researchers have proposed fast attention algorithms for LLMs by approximating or replace softmax attention with other functions, such as low-degree po…