Multi-Layer Relation Knowledge Distillation For Fingerprint Restoration

Abstract

Knowledge distillation involves a lightweight student model learning from the high-performance teacher model. Traditional feature-based knowledge distillation has limitations as it makes the student model mimic the teacher’s features. This approach lacks flexibility, particularly when there are significant architectural differences between the teacher and student models. In this paper, we introduced a multi-layer relation knowledge distillation (MRKD). MRKD focuses on learning the similarity between input patches of the teacher model at different layers. We also utilize the Attention-based Fusion (ABF) module to learn the optimal ratios between different layers, enabling the acquisition of information from multiple layers. We designed an asymmetric one-encoder-two-decoder lightweight deep learning model to restore fingerprint quality, aiming for sub-0.1-second inference times. Compared to the teacher model, the lightweight model achieves 27x faster inference time and 10x fewer parameters. The lightweight model achieves average 50% improvement in equal error rate(EER) on the FVC2002 and FVC2004 datasets compared to state-of-the-art approaches.