Align or attend? Toward More Efficient and Accurate Spoken Word Discovery Using Speech-to-Image Retrieval

Liming Wang; Xinsheng Wang; Mark Hasegawa-Johnson; Odette Scharenborg; Najim Dehak

ICASSP 2021accepted0 citations

Align or attend? Toward More Efficient and Accurate Spoken Word Discovery Using Speech-to-Image Retrieval

Liming Wang, Xinsheng Wang, Mark Hasegawa-Johnson, Odette Scharenborg, Najim Dehak

Abstract

Multimodal word discovery (MWD) is often treated as a byproduct of the speech-to-image retrieval problem. However, our theoretical analysis shows that some kind of alignment/attention mechanism is crucial for a MWD system to learn meaningful word-level representation. We verify our theory by conducting retrieval and word discovery experiments on MSCOCO and Flickr8k, and empirically demonstrate that both neural MT with self-attention and statistical MT achieve word discovery scores that are superior to those of a state-of-the-art neural retrieval system, outperforming it by 2% and 5% alignment F1 scores respectively.

BibTeX

@inproceedings{icassp2021_alignorattendtow,
  title = {Align or attend? Toward More Efficient and Accurate Spoken Word Discovery Using Speech-to-Image Retrieval},
  author = {Liming Wang and Xinsheng Wang and Mark Hasegawa-Johnson and Odette Scharenborg and Najim Dehak},
  booktitle = {ICASSP 2021},
  year = {2021}
}