AIConfPaper
← Search

Felix Dangel

17 accepted papers

2026

Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature

Angelo Porrello, Pietro Buzzega, Felix Dangel, Thomas Sommariva, Riccardo Salami, Lorenzo Bonicelli +1
ICLR 2026poster

Task Arithmetic yields a modular, scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors;…

Cited by 0Source ↗Scholar ↗
2026

Exploiting weight-space symmetries for approximating curvature

Artem Artemev, Rui Xia, Benjamin M. Boyd, Youjing Yu, Felix Dangel, Guillaume Hennequin +1
ICML 2026poster

Many machine learning techniques rely on approximating a loss function's curvature, but this is notoriously hard to do at the scale of modern deep networks. Surprisingly, no previous work has exploited the curvature constraints that arise from well known weight-space symmetries in loss landscapes. B…

Cited by 0Source ↗Scholar ↗
2026

Understanding and improving Shampoo and SOAP via Kullback-Leibler Minimization

Wu Lin, Scott C. Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, Roger Baker Grosse
ICLR 2026poster

Shampoo and its efficient, Adam-stabilized variant SOAP, employ structured second-moment estimation and have received growing attention for their effectiveness. In practice, Shampoo requires step-size grafting with Adam to achieve competitive performance. SOAP mitigates this by applying Adam in Sha…

Cited by 0Source ↗code ↗Scholar ↗
2025

Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator

Yu Xin Li, Felix Dangel, Derek Tam, Colin Raffel
ICML 2025spotlight

The diagonal of a model's Fisher Information Matrix (the "Fisher") has frequently been used as a way to measure parameter sensitivity. Typically, the Fisher is estimated by computing the squared gradient of the model's outputs with respect to its parameters, averaged over a few hundred or thousand e…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It

Marvin F. da Silva, Felix Dangel, Sageev Oore
ICML 2025spotlight

The concept of sharpness has been successfully applied to traditional architectures like MLPs and CNNs to predict their generalization. For transformers, however, recent work reported weak correlation between flatness and generalization. We argue that existing sharpness measures fail for transformer…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization

Andres Guzman-Cordero, Felix Dangel, Gil Goldshlager, Marius Zeinhofer
NeurIPS 2025poster

Natural gradient methods significantly accelerate the training of Physics-Informed Neural Networks (PINNs), but are often prohibitively costly. We introduce a suite of techniques to improve the accuracy and efficiency of energy natural gradient descent (ENGD) for PINNs. First, we leverage the Woodbu…

Cited by 0Source ↗Scholar ↗
2025

What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis

Weronika Ormaniec, Felix Dangel, Sidak Pal Singh
ICLR 2025spotlight

The Transformer architecture has inarguably revolutionized deep learning, overtaking classical architectures like multi-layer perceptions (MLPs) and convolutional neural networks (CNNs). At its core, the attention block differs in form and functionality from most other architectural components in de…

Cited by 4Source ↗PDF ↗code ↗Scholar ↗
2024

Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective

Wu Lin, Felix Dangel, Runa Eschenhagen, Juhan Bae, Richard E. Turner, Alireza Makhzani
ICML 2024poster

Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers. Their diagonal preconditioner is based on the gradient outer product which is incorporated into the parameter update via a square root. While these methods are of…

Cited by 12Source ↗PDF ↗Scholar ↗
2024

Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods

Felix Dangel
NeurIPS 2024poster

Despite their simple intuition, convolutions are more tedious to analyze than dense layers, which complicates the transfer of theoretical and algorithmic ideas to convolutions. We simplify convolutions by viewing them as tensor networks (TNs) that allow reasoning about the underlying tensor multipli…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks

Felix Dangel, Johannes Müller, Marius Zeinhofer
NeurIPS 2024poster

Physics-Informed Neural Networks (PINNs) are infamous for being hard to train. Recently, second-order methods based on natural gradient and Gauss-Newton methods have shown promising performance, improving the accuracy achieved by first-order methods by several orders of magnitude. While promising,…

Cited by 7Source ↗PDF ↗Scholar ↗
2024

Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning

Mohamed Elsayed, Homayoon Farrahi, Felix Dangel, A. Rupam Mahmood
ICML 2024poster

Second-order information is valuable for many applications but challenging to compute. Several works focus on computing or approximating Hessian diagonals, but even this simplification introduces significant additional costs compared to computing a gradient. In the absence of efficient exact computa…

Cited by 3Source ↗PDF ↗code ↗Scholar ↗
2024

Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC

Wu Lin, Felix Dangel, Runa Eschenhagen, Kirill Neklyudov, Agustinus Kristiadi, Richard E. Turner +1
ICML 2024poster

Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such meth…

Cited by 4Source ↗PDF ↗Scholar ↗
2023

The Geometry of Neural Nets' Parameter Spaces Under Reparametrization

Agustinus Kristiadi, Felix Dangel, Philipp Hennig
NeurIPS 2023spotlight

Model reparametrization, which follows the change-of-variable rule of calculus, is a popular way to improve the training of neural nets. But it can also be problematic since it can induce inconsistencies in, e.g., Hessian-based flatness measures, optimization trajectories, and modes of probability d…

Cited by 11Source ↗PDF ↗Scholar ↗
2021

Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks

Frank Schneider, Felix Dangel, Philipp Hennig
NeurIPS 2021poster

When engineers train deep learning models, they are very much "flying blind". Commonly used methods for real-time training diagnostics, such as monitoring the train/test loss, are limited. Assessing a network's training process solely through these performance indicators is akin to debugging softwar…

Cited by 13Source ↗PDF ↗Scholar ↗
2020

Modular Block-diagonal Curvature Approximations for Feedforward Architectures

Felix Dangel, Stefan Harmeling, Philipp Hennig
AISTATS 2020poster

We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual der…

Cited by 20Source ↗PDF ↗code ↗Scholar ↗