Frequency-Space Margin Perception for Open Set Knowledge Distillation

Abstract

Knowledge Distillation (KD) has attracted considerable attention as a typical model compression and knowledge transfer paradigm. However, most KD approaches are predicated on the implicit assumption: the deployed student model will exclusively encounter classes that have been seen during training, which goes against most real-world scenarios with unseen classes. To this end, we formalize the open-set KD task, where known classes are visible during training and unseen open-set classes appear when testing. We analyze the potential challenges of openset KD from the discriminability of embedding space and the semantic separability of label space. Then we propose a Frequency-Spatial Margin Perception (FSMP) framework composed of a frequency-spatial feature distillation paradigm and a semantic margin-based calibration strategy. Experiments on multiple benchmarks demonstrate that FSMP consistently improves both the closed-set and open-set performance of existing KD methods across homogeneous and heterogeneous frameworks.