Semi-Supervised Knowledge Distillation Framework towards Lightweight Large Language Model for Spoken Language Translation
Tonmoy Rajkhowa, Amartya Roy Chowdhury, Achyut Mani Tripathi, Sanjeev Sharma, Om Jee Pandey
Abstract
Even though large language models (LLMs) have demonstrated remarkable performance across various natural language processing tasks, their application in speech-related tasks has largely remained underexplored. This work addresses this gap by incorporating acoustic features into an LLM which can be fine-tuned for downstream direct speech-to-text translation and automatic speech recognition tasks. To address the computational demands associated with fine-tuning LLMs, a novel self and semi-supervised knowledge distillation technique is proposed to implement a lightweight LLM having 50% lesser parameters. Validated on the MuST-C and Librispeech datasets, this technique achieves over 92% of the performance of the larger LLM, demonstrating both robust performance and computational efficiency.
BibTeX
@inproceedings{icassp2025_semisupervisedkn,
title = {Semi-Supervised Knowledge Distillation Framework towards Lightweight Large Language Model for Spoken Language Translation},
author = {Tonmoy Rajkhowa and Amartya Roy Chowdhury and Achyut Mani Tripathi and Sanjeev Sharma and Om Jee Pandey},
booktitle = {ICASSP 2025},
year = {2025}
}