Christoph Hönes

NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents

While (large) language models have significantly improved over the last years, they still struggle to sensibly process long sequences found, e.g., in books, due to the quadratic scaling of the underlying attention mechanism. To address this, we propose NextLevelBERT, a Masked Language Model operatin…