wang ning

MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition

Linear attention offers the advantages of linear inference time and fixed memory usage compared to Softmax attention. However, training large-scale language models with linear attention from scratch remains prohibitively expensive and exhibits significant performance gaps compared to Softmax-based…