Karen Simonyan — accepted AI-conference papers

2022

An empirical analysis of compute-optimal large language model training

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford +16

NeurIPS 2022accept

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of trai…

Cited by 171 Source ↗PDF ↗Scholar ↗

2022

Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson +21

NeurIPS 2022accept

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bri…

Cited by 4376 Source ↗PDF ↗Scholar ↗

2022

Improving Language Models by Retrieving from Trillions of Tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican +22

ICML 2022spotlight

We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a 2 trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pil…

Cited by 1323 Source ↗PDF ↗code ↗Scholar ↗

2022

Unified Scaling Laws for Routed Language Models

Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann +20

ICML 2022oral

The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input. For these models, parameter c…

Cited by 79 Source ↗PDF ↗code ↗Scholar ↗

2021

End-to-end Adversarial Text-to-Speech

Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, Karen Simonyan

ICLR 2021oral

Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting…

Cited by 239 Source ↗PDF ↗Scholar ↗

2021

High-Performance Large-Scale Image Recognition Without Normalization

Andy Brock, Soham De, Samuel L Smith, Karen Simonyan

ICML 2021spotlight

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models…

Cited by 668 Source ↗PDF ↗code ↗Scholar ↗

2021

Machine Translation Decoding beyond Beam Search

Rémi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Lespiau Jean-Baptiste, Ioannis Antonoglou +2

EMNLP 2021main

Beam search is the go-to method for decoding auto-regressive machine translation models. While it yields consistent improvements in terms of BLEU, it is only concerned with finding outputs with high model likelihood, and is thus agnostic to whatever end metric or score practitioners care about. Our…

Cited by 73 Source ↗PDF ↗Scholar ↗

2021

Practical Real Time Recurrent Learning with a Sparse Approximation

Jacob Menick, Erich Elsen, Utku Evci, Simon Osindero, Karen Simonyan, Alex Graves

ICLR 2021spotlight

Recurrent neural networks are usually trained with backpropagation through time, which requires storing a complete history of network states, and prohibits updating the weights "online" (after every timestep). Real Time Recurrent Learning (RTRL) eliminates the need for history storage and allows for…

Cited by 22 Source ↗PDF ↗Scholar ↗

2020

Evolving Normalization-Activation Layers

Hanxiao Liu, Andy Brock, Karen Simonyan, Quoc V. Le

NeurIPS 2020spotlight

Normalization layers and activation functions are fundamental components in deep networks and typically co-locate with each other. Here we propose to design them using an automated approach. Instead of designing them separately, we unify them into a single tensor-to-tensor computation graph, and evo…

Cited by 111 Source ↗PDF ↗Scholar ↗

2020

Fast Sparse ConvNets

Erich Elsen, Marat Dukhan, Trevor Gale, Karen Simonyan

CVPR 2020poster

Historically, the pursuit of efficient inference has been one of the driving forces behind the research into new deep learning architectures and building blocks. Some of the recent examples include: the squeeze-and-excitation module, depthwise separable convolutions in Xception, and the inverted bot…

Cited by 176 PDF ↗Scholar ↗

2020

High Fidelity Speech Synthesis with Adversarial Networks

Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande +2

ICLR 2020talk

Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in the audio domain has received limited attention, and autoregressive models, such as WaveNet, remain the state of the ar…

Cited by 321 Source ↗code ↗Scholar ↗

2020

Off-Policy Actor-Critic with Shared Experience Replay

Simon Schmitt, Matteo Hessel, Karen Simonyan

ICML 2020poster

We investigate the combination of actor-critic reinforcement learning algorithms with a uniform large-scale experience replay and propose solutions for two ensuing challenges: (a) efficient actor-critic learning with experience replay (b) the stability of off-policy learning where agents learn from…

Cited by 63 Source ↗PDF ↗Scholar ↗

2019

DARTS: Differentiable Architecture Search

Hanxiao Liu, Karen Simonyan, Yiming Yang

ICLR 2019poster

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxati…

Cited by 5782 Source ↗PDF ↗code ↗Scholar ↗

2019

Large Scale Adversarial Representation Learning

Jeff Donahue, Karen Simonyan

NeurIPS 2019poster

Adversarially trained generative models (GANs) have recently achieved compelling image synthesis results. But despite early successes in using GANs for unsupervised representation learning, they have since been superseded by approaches based on self-supervision. In this work we show that progress in…

Cited by 641 Source ↗PDF ↗Scholar ↗

2019

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Andrew Brock, Jeff Donahue, Karen Simonyan

ICLR 2019oral

Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specifi…

Cited by 6860 Source ↗PDF ↗Scholar ↗

2018

Efficient Neural Audio Synthesis

Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart +4

ICML 2018oral

Sequential models achieve state-of-the-art results in audio, visual and textual domains with respect to both estimating the data distribution and generating desired samples. Efficient sampling for this class of models at the cost of little to no loss in quality has however remained an elusive proble…

Cited by 1097 Source ↗PDF ↗Scholar ↗

2018

Hierarchical Representations for Efficient Architecture Search

Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, Koray Kavukcuoglu

ICLR 2018poster

We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly…

Cited by 1205 Source ↗PDF ↗Scholar ↗

2018

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward +6

ICML 2018oral

In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Lear…

Cited by 1853 Source ↗PDF ↗code ↗Scholar ↗

2018

Learning to Navigate in Cities Without a Map

Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin +4

NeurIPS 2018poster

Navigating through unstructured environments is a basic capability of intelligent creatures, and thus is of fundamental interest in the study and development of artificial intelligence. Long-range navigation is a complex cognitive task that relies on developing an internal representation of space, g…

Cited by 363 Source ↗PDF ↗code ↗Scholar ↗

2018

Learning to search with MCTSnets

Arthur Guez, Theophane Weber, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals, Daan Wierstra +2

ICML 2018oral

Planning problems are among the most important and well-studied problems in artificial intelligence. They are most typically solved by tree search algorithms that simulate ahead into the future, evaluate future states, and back-up those evaluations to the root of a search tree. Among these algorithm…

Cited by 107 Source ↗PDF ↗Scholar ↗

2018

Parallel WaveNet: Fast High-Fidelity Speech Synthesis

Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu +16

ICML 2018oral

The recently-developed WaveNet architecture is the current state of the art in realistic speech synthesis, consistently rated as more natural sounding for many different languages than any previous system. However, because WaveNet relies on sequential generation of one audio sample at a time, it is…

Cited by 1053 Source ↗PDF ↗Scholar ↗

2018

The challenge of realistic music generation: modelling raw audio at scale

Sander Dieleman, Aaron van den Oord, Karen Simonyan

NeurIPS 2018poster

Realistic music generation is a challenging task. When building generative models of music that are learnt from data, typically high-level representations such as scores or MIDI are used that abstract away the idiosyncrasies of a particular performance. But these nuances are very important for our p…

Cited by 237 Source ↗PDF ↗Scholar ↗

2017

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck +1

ICML 2017poster

Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder…

Cited by 827 Source ↗PDF ↗Scholar ↗

2017

Video Pixel Networks

Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves +1

ICML 2017poster

We propose a probabilistic video model, the Video Pixel Network (VPN), that estimates the discrete joint distribution of the raw pixel values in a video. The model and the neural architecture reflect the time, space and color structure of video tensors and encode it as a four-dimensional dependency…

Cited by 528 Source ↗PDF ↗Scholar ↗

2015

Natural Neural Networks

Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, koray kavukcuoglu

NeurIPS 2015poster

We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the…

Cited by 241 Source ↗PDF ↗Scholar ↗

2015

Spatial Transformer Networks

Max Jaderberg, Karen Simonyan, Andrew Zisserman, koray kavukcuoglu

NeurIPS 2015spotlight

Convolutional Neural Networks define an exceptionallypowerful class of model, but are still limited by the lack of abilityto be spatially invariant to the input data in a computationally and parameterefficient manner. In this work we introduce a new learnable module, theSpatial Transformer, which ex…

Cited by 10066 Source ↗PDF ↗Scholar ↗