Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs
Vaggelis Dorovatas, Soroush Seifi, Gunshi Gupta, Rahaf Aljundi
Abstract
Video Large Language Models (Video-LLMs) excel at understanding videos in-context, assuming full access to the video when answering queries. However, these models face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. In this work, we propose a training-free approach compatible with standard Video-LLMs, leveraging three key concepts: 1) LLM-informed selection of visual tokens to identify those that the LLM has attended to and contributed to its understanding of each short clip. Our attention-based selection allows us to discard up to ~95\% of unimportant visual tokens with minimal performance loss; 2) Hierarchical selection of tokens combined with natural language understanding of each processed clip; 3) Caption-based question answering for lightweight and accurate responses. Our method achieves state-of-the-art performance on streaming video benchmarks, striking a balance between efficiency and effectiveness.
BibTeX
@inproceedings{
dorovatas2025recurrent,
title={Recurrent Attention-based Token Selection for Efficient Streaming Video-{LLM}s},
author={Vaggelis Dorovatas and Soroush Seifi and Gunshi Gupta and Rahaf Aljundi},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=1OuhWYrwgW}
}