Glenn M Fung — accepted AI-conference papers

2022

Multi Resolution Analysis (MRA) for Approximate Self-Attention

Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, Vikas Singh

ICML 2022spotlight

Transformers have emerged as a preferred model for many tasks in natural langugage processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effec…

Cited by 12 Source ↗PDF ↗code ↗Scholar ↗

2021

You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling

Zhanpeng Zeng, Yunyang Xiong, Sathya Ravi, Shailesh Acharya, Glenn M Fung, Vikas Singh

ICML 2021spotlight

Transformer-based models are widely used in natural language processing (NLP). Central to the transformer model is the self-attention mechanism, which captures the interactions of token pairs in the input sequences and depends quadratically on the sequence length. Training such models on longer sequ…

Cited by 27 Source ↗PDF ↗code ↗Scholar ↗