Sub-Band Contrastive Learning-Based Knowledge Distillation For Sound Classification

Abstract

Knowledge distillation(KD) technique is widely known for its outstanding ability to train a compact student model under supervision of a cumbersome pre-trained teacher network. The traditional KD technique focuses only on distilling dark knowledge using logits of a teacher network while neglecting the information regarding contrastive representation. To this end, we propose a new KD loss function that enables a student network to learn informative contrastive distribution and fine grained information from spectrogram representation of a signal thus enhancing performance of a student network for sound classification task. The experiments are conducted on two benchmark sound classification datasets, viz. ESC-10 and Audio MNIST, which illustrates that the student network trained using the proposed KD loss function outperformed the competitive KD techniques.