AIConfPaper
← Search

Dan Busbridge

14 accepted papers

2026

Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration

Bruno Kacper Mlodozeniec, Pierre Ablin, Louis Béthune, Dan Busbridge, Michal Klein, Jason Ramapuram +1
ICLR 2026poster

Hyperparameter tuning can dramatically impact training stability of large-scale models. Recent works on neural network parameterisations, such as μP, have shown that layer types and sizes should dictate how global hyperparameters should be rescaled in order to achieve efficient transfer across mode…

Cited by 0Source ↗Scholar ↗
2026

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training

Jakub Krajewski, Amitis Shidani, Dan Busbridge, Jason Ramapuram, Sam Wiseman
ICLR 2026poster

While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliable. This paper challenges that view by proposing a direct framework to model the scaling of downstream accuracy from the…

Cited by 0Source ↗code ↗Scholar ↗
2025

Distillation Scaling Laws

Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, Russell Webb
ICML 2025poster

We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and stud…

Cited by 52Source ↗PDF ↗Scholar ↗
2025

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin El-Nouby, Joshua M. Susskind, Vimal Thilak
ICML 2025poster

Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily defined by two dimensions: the number of model parameters and the compute per example. While scaling typically involves increa…

Cited by 6Source ↗PDF ↗Scholar ↗
2025

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

Louis Béthune, David Grangier, Dan Busbridge, Eleonora Gualdoni, marco cuturi, Pierre Ablin
ICML 2025poster

A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from that target domain. Finetuning presents two challenges: \textit{(i)} if the amount of target data is limited, as in most…

Cited by 1Source ↗PDF ↗Scholar ↗
2025

Scaling Laws for Optimal Data Mixtures

Mustafa Shukor, Louis Béthune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby +1
NeurIPS 2025poster

Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance. The standard approach to selecting this mixture relies on trial and error, which becomes impractical for large-scale p…

Cited by 0Source ↗Scholar ↗
2025

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

Jason Ramapuram, Federico Danieli, Eeshan Gunesh Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin +5
ICLR 2025poster

Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are typically obtained as the softmax of dot products between keys and queries. Recent work has explored alternatives to soft…

Cited by 11Source ↗PDF ↗code ↗Scholar ↗
2024

Poly-View Contrastive Learning

Amitis Shidani, R Devon Hjelm, Jason Ramapuram, Russell Webb, Eeshan Gunesh Dhekane, Dan Busbridge
ICLR 2024poster

Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investigate matching when there are more than two related views which we call poly-view tasks, and derive new representation lea…

Cited by 5Source ↗PDF ↗Scholar ↗
2023

DUET: 2D Structured and Approximately Equivariant Representations

Xavier Suau, Federico Danieli, T. Anderson Keller, Arno Blaas, Chen Huang, Jason Ramapuram +2
ICML 2023poster

Multiview Self-Supervised Learning (MSSL) is based on learning invariances with respect to a set of input transformations. However, invariance partially or totally removes transformation-related information from the representations, which might harm performance for specific downstream tasks that req…

Cited by 6Source ↗PDF ↗code ↗Scholar ↗
2023

How to Scale Your EMA

Dan Busbridge, Jason Ramapuram, Pierre Ablin, Tatiana Likhomanenko, Eeshan Gunesh Dhekane, Xavier Suau +1
NeurIPS 2023spotlight

Preserving training dynamics across batch sizes is an important tool for practical machine learning as it enables the trade-off between batch size and wall-clock time. This trade-off is typically enabled by a scaling rule, for example, in stochastic gradient descent, one should scale the learning ra…

Cited by 30Source ↗PDF ↗Scholar ↗
2023

Stabilizing Transformer Training by Preventing Attention Entropy Collapse

Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang +2
ICML 2023poster

Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for…

Cited by 71Source ↗PDF ↗code ↗Scholar ↗
2023

The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning

Borja Rodríguez Gálvez, Arno Blaas, Pau Rodriguez, Adam Golinski, Xavier Suau, Jason Ramapuram +2
ICML 2023poster

The mechanisms behind the success of multi-view self-supervised learning (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been studied through the lens of InfoNCE, a lower bound of the Mutual Information (MI). However, the relation between other MVSSL methods and MI remains uncle…

Cited by 13Source ↗PDF ↗code ↗Scholar ↗
2022

Position Prediction as an Effective Pretraining Strategy

Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram, Dan Busbridge, Tatiana Likhomanenko, Joseph Y Cheng +4
ICML 2022spotlight

Transformers \cite{transformer} have gained increasing popularity in a wide range of applications, including Natural Language Processing (NLP), Computer Vision and Speech Recognition, because of their powerful representational capacity. However, harnessing this representational capacity effectively…

Cited by 23Source ↗PDF ↗code ↗Scholar ↗
2018

Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks

Vitalii Zhelezniak, Dan Busbridge, April Shen, Samuel L. Smith, Nils Y. Hammerla
ICLR 2018workshop

Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks. Introducing the concept of an optimal representation space, we provide a simple theoretical resolution to this apparent paradox. In addition, we present a straightforward proced…

Cited by 8Source ↗code ↗Scholar ↗