Dynamic Video Frame Interpolation with Integrated Difficulty Pre-Assessment

Abstract

Video frame interpolation (VFI) has witnessed great progress in recent years. However, existing VFI models still struggle to achieve a good trade-off between accuracy and efficiency. Accurate VFI models typically rely on heavy compute to process all samples, ignoring the fact that easy samples with small motion or clear texture can be well addressed by a fast VFI model and do not require such heavy compute. In this paper, we present a dynamic VFI pipeline with integrated pre-assessment of interpolation difficulty. Specifically, it leverages a difficulty pre-assessment model to measure the difficulty level of interpolating input frames, and then dynamically selects an accurate or a fast VFI model for frame interpolation. Furthermore, we contribute a large-scale annotated dataset to train our VFI difficulty pre-assessment model. Extensive experiments show that our dynamic VFI pipeline can achieve an excellent trade-off between accuracy and efficiency, by feeding hard samples to accurate model, and passing easy samples through fast model.