A Time-Restricted Self-Attention Layer for ASR

Daniel Povey; Hossein Hadian; Pegah Ghahremani; Ke Li; Sanjeev Khudanpur

ICASSP 2018accepted0 citations

A Time-Restricted Self-Attention Layer for ASR

Daniel Povey, Hossein Hadian, Pegah Ghahremani, Ke Li, Sanjeev Khudanpur

Abstract

Self-attention - an attention mechanism where the input and output sequence lengths are the same - has recently been successfully applied to machine translation, caption generation, and phoneme recognition. In this paper we apply a restricted self-attention mechanism (with multiple heads) to speech recognition. By “restricted” we mean that the mechanism at a particular frame only sees input from a limited number of frames to the left and right. Restricting the context makes it easier to encode the position of the input - we use a I-hot encoding of the frame offset. We try introducing attention layers into TDNN architectures, and replacing LSTM layers with attention layers in TDNN+LSTM architectures. We show experiments on a number of ASR setups. We observe improvements compared to the TDNN and TDNN+LSTM baselines. Attention layers are also faster than LSTM layers in test time, since they lack recurrence.

BibTeX

@inproceedings{icassp2018_atimerestricteds,
  title = {A Time-Restricted Self-Attention Layer for ASR},
  author = {Daniel Povey and Hossein Hadian and Pegah Ghahremani and Ke Li and Sanjeev Khudanpur},
  booktitle = {ICASSP 2018},
  year = {2018}
}