AIConfPaper
← Search

Julien Launay

6 accepted papers

2023

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli +3
NeurIPS 2023poster

Large language models are commonly trained on a mixture of filtered web data and curated ``high-quality'' corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilit…

Cited by 147Source ↗PDF ↗Scholar ↗
2022

Adversarial Robustness by Design Through Analog Computing And Synthetic Gradients

Alessandro Cappelli, Ruben Ohana, Julien Launay, Laurent Meunier, Iacopo Poli, Florent Krzakala
ICASSP 2022accepted

We propose a new defense mechanism against adversarial at-tacks inspired by an optical co-processor, providing robustness without compromising natural accuracy in both white-box and black-box settings. This hardware co-processor performs a nonlinear fixed random transformation, where the parameters…

Cited by 0Source ↗Scholar ↗
2022

What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization?

Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy +2
ICML 2022spotlight

Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pretraining objectives used across state-of-the-art models differ significantly, an…

Cited by 214Source ↗PDF ↗code ↗Scholar ↗
2022

What Language Model to Train if You Have One Million GPU Hours?

Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman +12
EMNLP 2022finding

The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners. Simple, well-motivated architectural variations can transfer across tasks and scale, increasing the impact of modeling research. However, with the emergence of state-of-the-art 100B+ parame…

Cited by 124Source ↗PDF ↗code ↗Scholar ↗
2021

Photonic Differential Privacy with Direct Feedback Alignment

Ruben Ohana, Hamlet Jesse Medina Ruiz, Julien Launay, Alessandro Cappelli, Iacopo Poli, Liva Ralaivola +1
NeurIPS 2021poster

Optical Processing Units (OPUs) -- low-power photonic chips dedicated to large scale random projections -- have been used in previous work to train deep neural networks using Direct Feedback Alignment (DFA), an effective alternative to backpropagation. Here, we demonstrate how to leverage the intrin…

Cited by 13Source ↗PDF ↗Scholar ↗
2020

Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures

Julien Launay, Iacopo Poli, François Boniface, Florent Krzakala
NeurIPS 2020poster

Despite being the workhorse of deep learning, the backpropagation algorithm is no panacea. It enforces sequential layer updates, thus preventing efficient parallelization of the training process. Furthermore, its biological plausibility is being challenged. Alternative schemes have been devised; yet…

Cited by 96Source ↗PDF ↗code ↗Scholar ↗