Fast-Slow Transformer for Visually Grounding Speech

Puyuan Peng; David Harwath

ICASSP 2022accepted0 citations

Fast-Slow Transformer for Visually Grounding Speech

Abstract

We present Fast-Slow Transformer for Visually Grounding Speech, or FaST-VGS. FaST-VGS is a Transformer-based model for learning the associations between raw speech waveforms and visual images. The model unifies dual-encoder and cross-attention architectures into a single model, reaping the superior retrieval speed of the former along with the accuracy of the latter. FaST-VGS achieves state-of-the-art speech-image retrieval accuracy on benchmark datasets, and its learned representations exhibit strong performance on the ZeroSpeech 2021 phonetic and semantic tasks.

BibTeX

@inproceedings{icassp2022_fastslowtransfor,
  title = {Fast-Slow Transformer for Visually Grounding Speech},
  author = {Puyuan Peng and David Harwath},
  booktitle = {ICASSP 2022},
  year = {2022}
}