Ahmet Enis Cetin

Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers

Self-attention is central to the success of Transformer architectures; however, learning the query, key, and value projections from random initialization remains challenging and computationally expensive. In this paper, we propose two complementary methods that leverage the Discrete Cosine Transform