Arthur Mensch — accepted AI-conference papers

2022

An empirical analysis of compute-optimal large language model training

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford +16

NeurIPS 2022accept

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of trai…

Cited by 171 Source ↗PDF ↗Scholar ↗

2022

Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson +21

NeurIPS 2022accept

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bri…

Cited by 4376 Source ↗PDF ↗Scholar ↗

2022

Improving Language Models by Retrieving from Trillions of Tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican +22

ICML 2022spotlight

We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a 2 trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pil…

Cited by 1323 Source ↗PDF ↗code ↗Scholar ↗

2022

Unified Scaling Laws for Routed Language Models

Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann +20

ICML 2022oral

The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input. For these models, parameter c…

Cited by 79 Source ↗PDF ↗code ↗Scholar ↗

2021

Differentiable Divergences Between Time Series

Mathieu Blondel, Arthur Mensch, Jean-Philippe Vert

AISTATS 2021poster

Computing the discrepancy between time series of variable sizes is notoriously challenging. While dynamic time warping (DTW) is popularly used for this purpose, it is not differentiable everywhere and is known to lead to bad local optima when used as a “loss”. Soft-DTW addresses these issues, but it…

Cited by 72 Source ↗PDF ↗code ↗Scholar ↗

2020

A mean-field analysis of two-player zero-sum games

Carles Domingo-Enrich, Samy Jelassi, Arthur Mensch, Grant Rotskoff, Joan Bruna

NeurIPS 2020poster

Finding Nash equilibria in two-player zero-sum continuous games is a central problem in machine learning, e.g. for training both GANs and robust models. The existence of pure Nash equilibria requires strong conditions which are not typically met in practice. Mixed Nash equilibria exist in greater ge…

Cited by 66 Source ↗PDF ↗Scholar ↗

2020

Extra-gradient with player sampling for faster convergence in n-player games

Samy Jelassi, Carles Domingo-Enrich, Damien Scieur, Arthur Mensch, Joan Bruna

ICML 2020poster

Data-driven modeling increasingly requires to find a Nash equilibrium in multi-player games, e.g. when training GANs. In this paper, we analyse a new extra-gradient method for Nash equilibrium finding, that performs gradient extrapolations and updates on a random subset of players at each iteration.…

Cited by 4 Source ↗PDF ↗Scholar ↗

2020

Online Sinkhorn: Optimal Transport distances from sample streams

Arthur Mensch, Gabriel Peyré

NeurIPS 2020oral

Optimal Transport (OT) distances are now routinely used as loss functions in ML tasks. Yet, computing OT distances between arbitrary (i.e. not necessarily discrete) probability distributions remains an open problem. This paper introduces a new online estimator of entropy-regularized OT distances bet…

Cited by 39 Source ↗PDF ↗Scholar ↗

2019

Geometric Losses for Distributional Learning

Arthur Mensch, Mathieu Blondel, Gabriel Peyré

ICML 2019oral

Building upon recent advances in entropy-regularized optimal transport, and upon Fenchel duality between measures and continuous functions, we propose a generalization of the logistic loss that incorporates a metric or cost between classes. Unlike previous attempts to use optimal transport distances…

Cited by 22 Source ↗PDF ↗code ↗Scholar ↗

2018

Differentiable Dynamic Programming for Structured Prediction and Attention

Arthur Mensch, Mathieu Blondel

ICML 2018oral

Dynamic programming (DP) solves a variety of structured combinatorial problems by iteratively breaking them down into smaller subproblems. In spite of their versatility, many DP algorithms are non-differentiable, which hampers their use as a layer in neural networks trained by backpropagation. To ad…

Cited by 170 Source ↗PDF ↗Scholar ↗

2017

Learning Neural Representations of Human Cognition across Many fMRI Studies

Arthur Mensch, Julien Mairal, Danilo Bzdok, Bertrand Thirion, Gael Varoquaux

NeurIPS 2017poster

Cognitive neuroscience is enjoying rapid increase in extensive public brain-imaging datasets. It opens the door to large-scale statistical models. Finding a unified perspective for all available data calls for scalable and automated solutions to an old challenge: how to aggregate heterogeneous infor…

Cited by 47 Source ↗PDF ↗code ↗Scholar ↗

2016

Dictionary Learning for Massive Matrix Factorization

Arthur Mensch, Julien Mairal, Bertrand Thirion, Gael Varoquaux

ICML 2016poster

Sparse matrix factorization is a popular tool to obtain interpretable data decompositions, which are also effective to perform data completion or denoising. Its applicability to large datasets has been addressed with online and randomized methods, that reduce the complexity in one of the matrix dime…

Cited by 69 Source ↗PDF ↗code ↗Scholar ↗

2016

Learning brain regions via large-scale online structured sparse dictionary learning

Elvis DOHMATOB, Arthur Mensch, Gael Varoquaux, Bertrand Thirion

NeurIPS 2016poster

We propose a multivariate online dictionary-learning method for obtaining decompositions of brain images with structured and sparse components (aka atoms). Sparsity is to be understood in the usual sense: the dictionary atoms are constrained to contain mostly zeros. This is imposed via an $\ell_1$-n…

Cited by 23 Source ↗PDF ↗Scholar ↗