Sayak Chakrabarti

Poly-attention: a general scheme for higher-order self-attention

The self-attention mechanism, at the heart of the transformer model, is able to effectively model pairwise interactions between tokens. However, numerous recent works have shown that it is unable to perform basic tasks involving detecting triples of correlated tokens, or compositional tasks where mu…