Enhancing EEG-based Covert Speech Decoding through Knowledge Transfer

Abstract

Covert speech, the imagination of articulation without any actual movement of vocal apparatus, can aid individuals with speech impairments. Recent studies have shown the possibilities of decoding covert speech from non-invasive techniques such as electroencephalogram (EEG). Decoding covert speech from EEG may find broader applications than invasive approaches, but it poses additional challenges due to its less distinct speech-related brain patterns compared with overt speech. In this paper, we propose a novel knowledge transfer framework to build a pretrained model for covert speech from overt speech, including both EEG and audio data. We further integrate acoustic features to enforce a direct neural-to-phonetic mapping from EEG signals to audio representations using data from other subjects. Finally, we fine-tune the pretrained model with covert EEG from the target subjects for decoding. To validate the proposed strategy, we test it on our covert/overt EEG dataset collected from 54 subjects with concurrent EEG and audio recording. Results show that the proposed framework outperforms SOTA methods in terms of classification accuracy of covert speech decoding using EEG.