Ruslan Khalitov — accepted AI-conference papers

2023

ChordMixer: A Scalable Neural Attention Model for Sequences with Different Length

Ruslan Khalitov, Tong Yu, Lei Cheng, Zhirong Yang

ICLR 2023poster

Sequential data naturally have different lengths in many domains, with some very long sequences. As an important modeling tool, neural attention should capture long-range interaction in such sequences. However, most existing neural attention models admit only short sequences, or they have to employ…

Cited by 22 Source ↗PDF ↗Scholar ↗

2022

Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better Than Dot-Product Self-Attention

Tong Yu, Ruslan Khalitov, Lei Cheng, Zhirong Yang

CVPR 2022poster

Self-Attention is a widely used building block in neural modeling to mix long-range data elements. Most self-attention neural networks employ pairwise dot-products to specify the attention coefficients. However, these methods require O(N^2) computing cost for sequence length N. Even though some appr…

Cited by 14 PDF ↗code ↗Scholar ↗