2022
LittleBird: Efficient Faster & Longer Transformer for Question Answering
EMNLP 2022main
BERT has shown a lot of sucess in a wide variety of NLP tasks. But it has a limitation dealing with long inputs due to its attention mechanism. Longformer, ETC and BigBird addressed this issue and effectively solved the quadratic dependency problem.However we find that these models are not sufficien…