Gergely Szilvasy — accepted AI-conference papers

2026

Short Window Attention Enables Long-Term Memorization

Loïc Cabannes, Maximilian Beck, Maria Lomeli, Gergely Szilvasy, Matthijs Douze, Jade Copet +3

ICLR 2026poster

Recent works show that hybrid architectures combining sliding window softmax attention layers with linear recurrent neural network (RNN) layers outperform both of these architectures taken separately. However, the impact of the window length and the interplay between softmax attention and linear RNN…

Cited by 0 Source ↗Scholar ↗

2024

RA-DIT: Retrieval-Augmented Dual Instruction Tuning

Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James +6

ICLR 2024poster

Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the…

Cited by 142 Source ↗PDF ↗Scholar ↗