AIConfPaper
← Search

Jesse Engel

13 accepted papers

2022

General-purpose, long-context autoregressive modeling with Perceiver AR

Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski +9
ICML 2022spotlight

Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed…

Cited by 79Source ↗PDF ↗code ↗Scholar ↗
2022

MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling

Yusong Wu, Ethan Manilow, Yi Deng, Rigel Swavely, Kyle Kastner, Tim Cooijmans +3
ICLR 2022oral

Musical expression requires control of both what notes that are played, and how they are performed. Conventional audio synthesizers provide detailed expressive controls, but at the cost of realism. Black-box neural audio synthesis and concatenative samplers can produce realistic audio, but have few…

Cited by 67Source ↗PDF ↗code ↗Scholar ↗
2022

MT3: Multi-Task Multitrack Music Transcription

Joshua P Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, Jesse Engel
ICLR 2022spotlight

Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneou…

Cited by 126Source ↗PDF ↗code ↗Scholar ↗
2020

DDSP: Differentiable Digital Signal Processing

Jesse Engel, Lamtharn (Hanoi) Hantrakul, Chenjie Gu, Adam Roberts
ICLR 2020spotlight

Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizer…

Cited by 562Source ↗code ↗Scholar ↗
2020

Encoding Musical Style with Transformer Autoencoders

Kristy Choi, Curtis Hawthorne, Ian Simon, Monica Dinculescu, Jesse Engel
ICML 2020poster

We consider the problem of learning high-level controls over the global structure of generated sequences, particularly in the context of symbolic music generation with complex language models. In this work, we present the Transformer autoencoder, which aggregates encodings of the input data across t…

Cited by 136Source ↗PDF ↗Scholar ↗
2019

Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman +3
ICLR 2019oral

Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we s…

Cited by 629Source ↗PDF ↗Scholar ↗
2019

GANSynth: Adversarial Neural Audio Synthesis

Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, Adam Roberts
ICLR 2019poster

Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling…

Cited by 604Source ↗PDF ↗Scholar ↗
2019

Learning to Groove with Inverse Sequence Transformations

Jon Gillick, Adam Roberts, Jesse Engel, Douglas Eck, David Bamman
ICML 2019oral

We explore models for translating abstract musical ideas (scores, rhythms) into expressive performances using seq2seq and recurrent variational information bottleneck (VIB) models. Though seq2seq models usually require painstakingly aligned corpora, we show that it is possible to adapt an approach f…

Cited by 136Source ↗PDF ↗Scholar ↗
2018

A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music

Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, Douglas Eck
ICML 2018oral

The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling se…

Cited by 676Source ↗PDF ↗Scholar ↗
2018

Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models

Jesse Engel, Matthew Hoffman, Adam Roberts
ICLR 2018poster

Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions. Conditional generation enables interactive control, but creating new controls often requires expensive retraining. In this paper, we develop a method to condition gener…

Cited by 171Source ↗PDF ↗Scholar ↗
2017

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck +1
ICML 2017poster

Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder…

Cited by 827Source ↗PDF ↗Scholar ↗
2016

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin

Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case +63
ICML 2016poster

We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech–two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of s…

Cited by 4031Source ↗PDF ↗code ↗Scholar ↗
2016

Persistent RNNs: Stashing Recurrent Weights On-Chip

Greg Diamos, Shubho Sengupta, Bryan Catanzaro, Mike Chrzanowski, Adam Coates, Erich Elsen +3
ICML 2016poster

This paper introduces a new technique for mapping Deep Recurrent Neural Networks (RNN) efficiently onto GPUs. We show how it is possi- ble to achieve substantially higher computational throughput at low mini-batch sizes than direct implementations of RNNs based on matrix multiplications. The key to…

Cited by 126Source ↗PDF ↗Scholar ↗