Roy Sheffer

I Hear Your True Colors: Image Guided Audio Generation

We propose IM2WAV, an image guided open-domain audio generation system. Given an input image or a sequence of images, IM2WAV generates a semantically relevant sound. IM2WAV is based on two Transformer language models, that operate over a hierarchical discrete audio representation obtained from a VQ-…