Dementia Detection by Fusing Speech and Eye-Tracking Representation

Zhengyan Sheng; Zhiqiang Guo; Xin Li; Yunxia Li; Zhenhua Ling

ICASSP 2022accepted0 citations

Dementia Detection by Fusing Speech and Eye-Tracking Representation

Zhengyan Sheng, Zhiqiang Guo, Xin Li, Yunxia Li, Zhenhua Ling

Abstract

This paper proposes a method of detecting dementia from the simultaneous speech and eye-tracking recordings of subjects in a picture description task. First, automatic speech recognition (ASR) and regional picture recognition (RPR) models are built to extract content-related bottleneck (BN) features for both speech and eye-tracking inputs. Then, a neural network is designed to fuse these two modals for discriminating dementia patients from healthy controls. The network contains a cross-modal Transformer encoder for bimodal interaction, and a self-attention Transformer encoder for final classification. Experimental results demonstrate that the detection accuracy of the proposed method is 84.26%, which outperforms baseline methods and ablated models using single speech or eye-tracking input.

BibTeX

@inproceedings{icassp2022_dementiadetectio,
  title = {Dementia Detection by Fusing Speech and Eye-Tracking Representation},
  author = {Zhengyan Sheng and Zhiqiang Guo and Xin Li and Yunxia Li and Zhenhua Ling},
  booktitle = {ICASSP 2022},
  year = {2022}
}