AIConfPaper
← Search

David Lopez-Paz

25 accepted papers

2026

Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries

Divyat Mahajan, Sachin Goyal, Badr Youbi Idrissi, Mohammad Pezeshki, Ioannis Mitliagkas, David Lopez-Paz +1
ICLR 2026poster

Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitations largely attributed to teacher-forced training. Multi-token prediction (MTP) partially mitigates these issues by pre…

Cited by 0Source ↗Scholar ↗
2026

Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling

Sachin Goyal, David Lopez-Paz, Kartik Ahuja
ICLR 2026poster

In the past year, distillation has seen a renewed prominence in large language model (LLM) pretraining, exemplified by the Llama-3.2 and Gemma model families. While distillation has historically been shown to improve statistical modeling, its effects on new paradigms key to modern LLMs—such as test-…

Cited by 0Source ↗Scholar ↗
2025

From Bytes to Ideas: Language Modeling with Autoregressive U-Nets

Mathurin VIDEAU, Badr Youbi Idrissi, Alessandro Leite, Marc Schoenauer, Olivier Teytaud, David Lopez-Paz
NeurIPS 2025poster

Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and similar schemes split text once, build a static vocabulary, and leave the model stuck with that choice. We relax this rigidit…

Cited by 0Source ↗code ↗Scholar ↗
2025

The Pitfalls of Memorization: When Memorization Hurts Generalization

Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz, Pascal Vincent
ICLR 2025poster

Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations. This behavior leads to poor generalization when the learned explanations rely on spurious correlations. In this work, we formalize $\textit{the interpla…

Cited by 3Source ↗PDF ↗code ↗Scholar ↗
2024

Better & Faster Large Language Models via Multi-token Prediction

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, Gabriel Synnaeve
ICML 2024poster

Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the mode…

Cited by 85Source ↗PDF ↗Scholar ↗
2024

Discovering Environments with XRM

Mohammad Pezeshki, Diane Bouchacourt, Mark Ibrahim, Nicolas Ballas, Pascal Vincent, David Lopez-Paz
ICML 2024oral

Environment annotations are essential for the success of many out-of-distribution (OOD) generalization methods. Unfortunately, these are costly to obtain and often limited by human annotators' biases. To achieve robust generalization, it is essential to develop algorithms for automatic environment d…

Cited by 24Source ↗PDF ↗code ↗Scholar ↗
2023

ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations

Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nicolas Ballas +4
ICLR 2023top-25%

Deep learning vision systems are widely deployed across applications where reliability is critical. However, even today's best models can fail to recognize an object when its pose, lighting, or background varies. While existing benchmarks surface examples challenging for models, they do not explain…

Cited by 50Source ↗PDF ↗Scholar ↗
2023

Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization

Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Leon Bottou, David Lopez-Paz
ICML 2023poster

Foundation models are redefining how AI systems are built. Practitioners now follow a standard procedure to build their machine learning solutions: from a pre-trained foundation model, they fine-tune the weights on the target task of interest. So, the Internet is swarmed by a handful of foundation m…

Cited by 78Source ↗PDF ↗code ↗Scholar ↗
2023

Why does Throwing Away Data Improve Worst-Group Error?

Kamalika Chaudhuri, Kartik Ahuja, Martin Arjovsky, David Lopez-Paz
ICML 2023oral

When facing data with imbalanced classes or groups, practitioners follow an intriguing strategy to achieve best results. They throw away examples until the classes or groups are balanced in size, and then perform empirical risk minimization on the reduced training set. This opposes common wisdom in…

Cited by 18Source ↗PDF ↗Scholar ↗
2022

Rich Feature Construction for the Optimization-Generalization Dilemma

Jianyu Zhang, David Lopez-Paz, Leon Bottou
ICML 2022spotlight

There often is a dilemma between ease of optimization and robust out-of-distribution (OoD) generalization. For instance, many OoD methods rely on penalty terms whose optimization is challenging. They are either too strong to optimize reliably or too weak to achieve their goals. We propose to initial…

Cited by 49Source ↗PDF ↗code ↗Scholar ↗
2021

An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers

Shanmukha Ramakrishna Vedantam, David Lopez-Paz, David J. Schwab
NeurIPS 2021poster

Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming specialized training algorithms for domain generalization. The goal of this paper is to further understand this phenomenon. In particular, we stud…

Cited by 51Source ↗PDF ↗Scholar ↗
2021

Using Hindsight to Anchor Past Knowledge in Continual Learning

Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, David Lopez-Paz
AAAI 2021technical

In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement diff…

Cited by 277Source ↗PDF ↗code ↗Scholar ↗
2020

Permutation Equivariant Models for Compositional Generalization in Language

Jonathan Gordon, David Lopez-Paz, Marco Baroni, Diane Bouchacourt
ICLR 2020poster

Humans understand novel sentences by composing meanings and roles of core language components. In contrast, neural network models for natural language modeling fail when such compositional generalization is required. The main contribution of this paper is to hypothesize that language compositionalit…

Cited by 129Source ↗code ↗Scholar ↗
2019

First-Order Adversarial Vulnerability of Neural Networks and Input Dimension

Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, David Lopez-Paz
ICML 2019oral

Over the past few years, neural networks were proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions. We show that adversarial vulnerability increases with the gradients of the training objective when viewed as a function of…

Cited by 130Source ↗PDF ↗code ↗Scholar ↗
2019

Learning about an exponential amount of conditional distributions

Mohamed Belghazi, Maxime Oquab, David Lopez-Paz
NeurIPS 2019poster

We introduce the Neural Conditioner (NC), a self-supervised machine able to learn about all the conditional distributions of a random vector X. The NC is a function NC(x⋅a,a,r) that leverages adversarial training to match each conditional distribution P(Xr|Xa=xa). After training, the NC generalizes…

Cited by 33Source ↗PDF ↗code ↗Scholar ↗
2019

Manifold Mixup: Better Representations by Interpolating Hidden States

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz +1
ICML 2019oral

Deep neural networks excel at learning the training data, but often provide incorrect and confident predictions when evaluated on slightly different test examples. This includes distribution shifts, outliers, and adversarial examples. To address these issues, we propose \manifoldmixup{}, a simple re…

Cited by 1641Source ↗PDF ↗code ↗Scholar ↗
2018

mixup: Beyond Empirical Risk Minimization

Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz
ICLR 2018poster

Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of…

Cited by 12891Source ↗PDF ↗code ↗Scholar ↗
2017

Discovering Causal Signals in Images

David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, Leon Bottou
CVPR 2017spotlight

This paper establishes the existence of observable footprints that reveal the "causal dispositions" of the object categories appearing in collections of images. We achieve this goal in two steps. First, we take a learning approach to observational causal discovery, and build a classifier that achi…

Cited by 302PDF ↗Scholar ↗
2015

Towards a Learning Theory of Cause-Effect Inference

David Lopez-Paz, Krikamol Muandet, Bernhard Schölkopf, Iliya Tolstikhin
ICML 2015poster

We pose causal inference as the problem of learning to classify probability distributions. In particular, we assume access to a collection {(S_i,l_i)}_i=1^n, where each S_i is a sample drawn from the probability distribution of X_i \times Y_i, and l_i is a binary label indicating whether “X_i \to Y_…

Cited by 229Source ↗PDF ↗code ↗Scholar ↗