AIConfPaper
← Search

Juhan Bae

14 accepted papers

2025

Accelerating neural network training: An analysis of the AlgoPerf competition

Priya Kasimbeg, Frank Schneider, Runa Eschenhagen, Juhan Bae, Chandramouli Shama Sastry, Mark Saroufim +8
ICLR 2025poster

The goal of the AlgoPerf: Training Algorithms competition is to evaluate practical speed-ups in neural network training achieved solely by improving the underlying training algorithms. In the external tuning ruleset, submissions must provide workload-agnostic hyperparameter search spaces, while in t…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Better Training Data Attribution via Better Inverse Hessian-Vector Products

Andrew Wang, Elisa Nguyen, Runshi Yang, Juhan Bae, Sheila A. McIlraith, Roger Baker Grosse
NeurIPS 2025poster

Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is dif…

Cited by 0Source ↗Scholar ↗
2025

IF-Guide: Influence Function-Guided Detoxification of LLMs

Zachary Coalson, Juhan Bae, Nicholas Carlini, Sanghyun Hong
NeurIPS 2025poster

We study how training data contributes to the emergence of toxic behaviors in large language models. Most prior work on reducing model toxicity adopts *reactive* approaches, such as fine-tuning pre-trained (and potentially toxic) models to align them with human values. In contrast, we propose a *pro…

Cited by 0Source ↗code ↗Scholar ↗
2025

Influence Functions for Scalable Data Attribution in Diffusion Models

Bruno Kacper Mlodozeniec, Runa Eschenhagen, Juhan Bae, Alexander Immer, David Krueger, Richard E. Turner
ICLR 2025oral

Diffusion models have led to significant advancements in generative modelling. Yet their widespread adoption poses challenges regarding data attribution and interpretability. In this paper, we aim to help address such challenges in diffusion models by extending influence functions. Influence functio…

Cited by 2Source ↗PDF ↗Scholar ↗
2025

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwaraknath Gnaneshwar, Acyr Locatelli +4
ICLR 2025poster

The capabilities and limitations of Large Language Models (LLMs) have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when…

Cited by 9Source ↗PDF ↗code ↗Scholar ↗
2025

What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions

Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Youngseog Chung, Adithya Pratapa +7
NeurIPS 2025poster

Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited. In response to this issue, data valuation (or data attribution), which quantifies the contribution or value of each data to the model output, has been discussed as a potential…

Cited by 0Source ↗code ↗Scholar ↗
2024

Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective

Wu Lin, Felix Dangel, Runa Eschenhagen, Juhan Bae, Richard E. Turner, Alireza Makhzani
ICML 2024poster

Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers. Their diagonal preconditioner is based on the gradient outer product which is incorporated into the parameter update via a square root. While these methods are of…

Cited by 12Source ↗PDF ↗Scholar ↗
2023

Efficient Parametric Approximations of Neural Network Function Space Distance

Nikita Dhawan, Sicong Huang, Juhan Bae, Roger Baker Grosse
ICML 2023poster

It is often useful to compactly summarize important properties of model parameters and training data so that they can be used later without storing and/or iterating over the entire dataset. As a specific case, we consider estimating the Function Space Distance (FSD) over a training set, i.e. the ave…

Cited by 6Source ↗PDF ↗Scholar ↗
2023

Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve

Juhan Bae, Michael R. Zhang, Michael Ruan, Eric Wang, So Hasegawa, Jimmy Ba +1
ICLR 2023top-5%

Variational autoencoders (VAEs) are powerful tools for learning latent representations of data used in a wide range of applications. In practice, VAEs usually require multiple training rounds to choose the amount of information the latent variable should retain. This trade-off between the reconstruc…

Cited by 21Source ↗PDF ↗Scholar ↗
2022

If Influence Functions are the Answer, Then What is the Question?

Juhan Bae, Nathan Hoyen Ng, Alston Lo, Marzyeh Ghassemi, Roger Baker Grosse
NeurIPS 2022accept

Influence functions efficiently estimate the effect of removing a single training data point on a model's learned parameters. While influence estimates align well with leave-one-out retraining for linear models, recent works have shown this alignment is often poor in neural networks. In this work, w…

Cited by 114Source ↗PDF ↗code ↗Scholar ↗
2021

On Monotonic Linear Interpolation of Neural Network Parameters

James R Lucas, Juhan Bae, Michael R Zhang, Stanislav Fort, Richard Zemel, Roger B Grosse
ICML 2021spotlight

Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective. This Monotonic Linear Interpolation (MLI) property, first observed by Goodfellow et al. 201…

Cited by 15Source ↗PDF ↗Scholar ↗
2020

Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians

Juhan Bae, Roger B Grosse
NeurIPS 2020poster

Hyperparameter optimization of neural networks can be elegantly formulated as a bilevel optimization problem. While research on bilevel optimization of neural networks has been dominated by implicit differentiation and unrolling, hypernetworks such as Self-Tuning Networks (STNs) have recently gained…

Cited by 28Source ↗PDF ↗code ↗Scholar ↗