HDMRAFT: Heterogeneous Distillation Matching-Recurrent all-Pairs Field Transform for Lightweight Optical Flow Estimation

Abstract

Optical flow can enable robots to accurately perceive the environment and support advanced applications. The estimation accuracy of directly pruned optical flow models suffers significant degradation and the resulting compact architectures are critically constrained by extreme operational scenarios, particularly those involving large displacements and occlusions. In this letter, we propose HDMRAFT, a robust and lightweight dense optical flow approach by combining structure optimization and compression techniques. We first implement Adaptability-Oriented Architectural Modifications (AOAM) before applying structured pruning, then jointly employ a Spatio-Temporal Feature Augmentation module (STFA) and a Global Matching Initialization Flow module (GMIF) to enhance motion representation learning. The resultant network derived from this optimized pipeline is established as the student network. Moreover, we discover that conventional knowledge distillation demonstrates limited efficacy when confronted with heterogeneous optical flow architectures and complex motion fields. To overcome this limitation, we present Heterogeneous Structured Knowledge Distillation (HSKD) that accounts for Transformer-aware components, enabling structured knowledge transfer between heterogeneous teacher-student optical flow models. We tested our approach on the optical flow benchmark datasets, Sintel and KITTI. Our approach achieves state-of-the-art performance at the same scale, while superior to the baseline model with an 81.2% reduction in parameters, an 82.1% decrease in computational complexity, and 2× speedup in inference.