AIConfPaper
← Search

Aurelien Lucchi

58 accepted papers

2026

Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective

Enea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov, Aurelien Lucchi, Anastasia Koloskova
ICLR 2026poster

Differential Privacy (DP) is becoming central to large-scale training as privacy regulations tighten. We revisit how DP noise interacts with *adaptivity* in optimization through the lens of *stochastic differential equations*, providing the first SDE-based analysis of private optimizers. Focusing on…

Cited by 0Source ↗code ↗Scholar ↗
2026

Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

Shyam Sundhar Ramesh, Xiaotong Ji, Matthieu Zimmer, Sangwoong Yoon, Zhiyong Wang, Haitham Bou Ammar +2
ICML 2026poster

RL-based post-training with GRPO is widely used to improve large language models on individual reasoning tasks. However, real-world deployment requires reliable performance across diverse tasks. A straightforward multi-task adaptation of GRPO often leads to imbalanced outcomes, with some tasks domin…

Cited by 0Source ↗Scholar ↗
2026

On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach

Enea Monzio Compagnoni, Rustem Islamov, Frank Proske, Aurelien Lucchi, Antonio Orvieto, Eduard Gorbunov
ICML 2026poster

Distributed stochastic optimization intertwines (i) stochastic gradient noise, (ii) communication compression, and (iii) adaptive/normalized updates. While each factor has been studied in isolation, their joint effect under realistic assumptions remains poorly understood. In this work, we develop a …

Cited by 0Source ↗Scholar ↗
2026

On the Role of Batch Size in Stochastic Conditional Gradient Methods

Rustem Islamov, Roman Machacek, Aurelien Lucchi, Antonio Silveti-Falls, Eduard Gorbunov, Volkan Cevher
ICML 2026poster

We study the role of batch size in stochastic conditional gradient methods under a $\mu$-Kurdyka–Łojasiewicz ($\mu$-KL) condition. Focusing on momentum-based stochastic Frank–Wolfe–type conditional gradient algorithms (e.g., Scion), we derive a new analysis that explicitly captures the interaction b…

Cited by 0Source ↗Scholar ↗
2026

Optimizer Choice Matters For The Emergence of Neural Collapse

Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk, Aurelien Lucchi
ICLR 2026poster

Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the op…

Cited by 0Source ↗Scholar ↗
2026

When Bias Helps Learning: Bridging Initial Prejudice and Trainability

Alberto Bassi, Marco Baity-Jesi, Aurelien Lucchi, Carlo Albert, Emanuele Francazi
ICLR 2026poster

Understanding the statistical properties of deep neural networks (DNNs) at initialization is crucial for elucidating both their trainability and the intrinsic architectural biases they encode prior to data exposure. Mean-field (MF) analyses have demonstrated that the parameter distribution in random…

Cited by 0Source ↗Scholar ↗
2025

Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise

Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, Aurelien Lucchi
ICLR 2025poster

Despite the vast empirical evidence supporting the efficacy of adaptive optimization methods in deep learning, their theoretical understanding is far from complete. This work introduces novel SDEs for commonly used adaptive optimizers: SignSGD, RMSprop(W), and Adam(W). These SDEs offer a quantitativ…

Cited by 2Source ↗PDF ↗Scholar ↗
2025

Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size

Rustem Islamov, Niccolò Ajroldi, Antonio Orvieto, Aurelien Lucchi
NeurIPS 2025poster

Modern optimization algorithms that incorporate momentum and adaptive step-size offer improved performance in numerous challenging deep learning tasks. However, their effectiveness is often highly sensitive to the choice of hyperparameters, especially the step-size. Tuning these parameters is often…

Cited by 0Source ↗Scholar ↗
2025

Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs

Enea Monzio Compagnoni, Rustem Islamov, Frank Norbert Proske, Aurelien Lucchi
AISTATS 2025oral

Distributed methods are essential for handling machine learning pipelines comprising large-scale models and datasets. However, their benefits often come at the cost of increased communication overhead between the central server and agents, which can become the main bottleneck, making training costly…

Cited by 0Source ↗Scholar ↗
2024

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression

Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, David Belius
NeurIPS 2024poster

This paper conducts a comprehensive study of the learning curves of kernel ridge regression (KRR) under minimal assumptions. Our contributions are three-fold: 1) we analyze the role of key properties of the kernel, such as its spectral eigen-decay, the characteristics of the eigenfunctions, and the…

Cited by 1Source ↗PDF ↗Scholar ↗
2024

Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum

Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, David Belius
ICML 2024poster

We derive new bounds for the condition number of kernel matrices, which we then use to enhance existing non-asymptotic test error bounds for kernel ridgeless regression in the over-parameterized regime for a fixed input dimension. For kernels with polynomial spectral decay, we recover the bound from…

Cited by 13Source ↗PDF ↗Scholar ↗
2024

Initial Guessing Bias: How Untrained Networks Favor Some Classes

Emanuele Francazi, Aurelien Lucchi, Marco Baity-Jesi
ICML 2024poster

Understanding and controlling biasing effects in neural networks is crucial for ensuring accurate and fair model performance. In the context of classification problems, we provide a theoretical analysis demonstrating that the structure of a deep neural network (DNN) can condition the model to assign…

Cited by 3Source ↗PDF ↗code ↗Scholar ↗
2024

Loss Landscape Characterization of Neural Networks without Over-Parametrization

Rustem Islamov, Niccolò Ajroldi, Antonio Orvieto, Aurelien Lucchi
NeurIPS 2024poster

Modern machine learning heavily depends on the effectiveness of optimization techniques. While deep learning models have achieved remarkable empirical results in training, their theoretical underpinnings remain somewhat elusive. Ensuring the convergence of optimization methods requires imposing spec…

Cited by 6Source ↗PDF ↗Scholar ↗
2024

SDEs for Minimax Optimization

Enea Monzio Compagnoni, Antonio Orvieto, Hans Kersting, Frank Proske, Aurelien Lucchi
AISTATS 2024poster

Minimax optimization problems have attracted a lot of attention over the past few years, with applications ranging from economics to machine learning. While advanced optimization methods exist for such problems, characterizing their dynamics in stochastic scenarios remains notably challenging. In th…

Cited by 4Source ↗PDF ↗code ↗Scholar ↗
2024

Theoretical Characterisation of the Gauss Newton Conditioning in Neural Networks

Jim Zhao, Sidak Pal Singh, Aurelien Lucchi
NeurIPS 2024poster

The Gauss-Newton (GN) matrix plays an important role in machine learning, most evident in its use as a preconditioning matrix for a wide family of popular adaptive methods to speed up optimization. Besides, it can also provide key insights into the optimization landscape of neural networks. In the…

Cited by 0Source ↗PDF ↗Scholar ↗
2023

A Theoretical Analysis of the Learning Dynamics under Class Imbalance

Emanuele Francazi, Marco Baity-Jesi, Aurelien Lucchi
ICML 2023poster

Data imbalance is a common problem in machine learning that can have a critical effect on the performance of a model. Various solutions exist but their impact on the convergence of the learning dynamics is not understood. Here, we elucidate the significant negative impact of data imbalance on learni…

Cited by 19Source ↗PDF ↗code ↗Scholar ↗
2023

A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regression

Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, Ivan Dokmanić, David Belius
NeurIPS 2023poster

Existing statistical learning guarantees for general kernel regressors often yield loose bounds when used with finite-rank kernels. Yet, finite-rank kernels naturally appear in a number of machine learning problems, e.g. when fine-tuning a pre-trained deep neural network's last layer to adapt it to…

Cited by 10Source ↗PDF ↗Scholar ↗
2023

An SDE for Modeling SAM: Theory and Insights

Enea Monzio Compagnoni, Luca Biggio, Antonio Orvieto, Frank Norbert Proske, Hans Kersting, Aurelien Lucchi
ICML 2023poster

We study the SAM (Sharpness-Aware Minimization) optimizer which has recently attracted a lot of interest due to its increased performance over more classical variants of stochastic gradient descent. Our main contribution is the derivation of continuous-time models (in the form of SDEs) for SAM and t…

Cited by 26Source ↗PDF ↗Scholar ↗
2023

Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers

Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, Thomas Hofmann
NeurIPS 2023spotlight

Autoregressive Transformers adopted in Large Language Models (LLMs) are hard to scale to long sequences. Despite several works trying to reduce their computational cost, most of LLMs still adopt attention layers between all pairs of tokens in the sequence, thus incurring a quadratic cost. In this st…

Cited by 62Source ↗PDF ↗Scholar ↗
2022

A Globally Convergent Evolutionary Strategy for Stochastic Constrained Optimization with Applications to Reinforcement Learning

Youssef Diouane, Aurelien Lucchi, Vihang Prakash Patil
AISTATS 2022poster

Evolutionary strategies have recently been shown to achieve competing levels of performance for complex optimization problems in reinforcement learning. In such problems, one often needs to optimize an objective function subject to a set of constraints, including for instance constraints on the entr…

Cited by 4Source ↗PDF ↗Scholar ↗
2022

Anticorrelated Noise Injection for Improved Generalization

Antonio Orvieto, Hans Kersting, Frank Proske, Francis Bach, Aurelien Lucchi
ICML 2022spotlight

Injecting artificial noise into gradient descent (GD) is commonly employed to improve the performance of machine learning models. Usually, uncorrelated noise is used in such perturbed gradient descent (PGD) methods. It is, however, not known if this is optimal or whether other types of noise could p…

Cited by 55Source ↗PDF ↗Scholar ↗
2022

Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity

Junchi Yang, Antonio Orvieto, Aurelien Lucchi, Niao He
AISTATS 2022poster

Gradient descent ascent (GDA), the simplest single-loop algorithm for nonconvex minimax optimization, is widely used in practical applications such as generative adversarial networks (GANs) and adversarial training. Albeit its desirable simplicity, recent work shows inferior convergence rates of GDA…

Cited by 77Source ↗PDF ↗code ↗Scholar ↗
2022

On the Theoretical Properties of Noise Correlation in Stochastic Optimization

Aurelien Lucchi, Frank Proske, Antonio Orvieto, Francis Bach, Hans Kersting
NeurIPS 2022accept

Studying the properties of stochastic noise to optimize complex non-convex functions has been an active area of research in the field of machine learning. Prior work~\citep{zhou2019pgd, wei2019noise} has shown that the noise of stochastic gradient descent improves optimization by overcoming undesira…

Cited by 9Source ↗PDF ↗Scholar ↗
2022

Phenomenology of Double Descent in Finite-Width Neural Networks

Sidak Pal Singh, Aurelien Lucchi, Thomas Hofmann, Bernhard Schölkopf
ICLR 2022poster

`Double descent' delineates the generalization behaviour of models depending on the regime they belong to: under- or over-parameterized. The current theoretical understanding behind the occurrence of this phenomenon is primarily based on linear and kernel regression models --- with informal parallel…

Cited by 13Source ↗PDF ↗Scholar ↗
2022

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse

Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, Aurelien Lucchi
NeurIPS 2022accept

Transformers have achieved remarkable success in several domains, ranging from natural language processing to computer vision. Nevertheless, it has been recently shown that stacking self-attention layers — the distinctive architectural component of Transformers — can result in rank collapse of the t…

Cited by 88Source ↗PDF ↗Scholar ↗
2022

Vanishing Curvature in Randomly Initialized Deep ReLU Networks

Antonio Orvieto, Jonas Kohler, Dario Pavllo, Thomas Hofmann, Aurelien Lucchi
AISTATS 2022poster

Deep ReLU networks are at the basis of many modern neural architectures. Yet, the loss landscape of such networks and its interaction with state-of-the-art optimizers is not fully understood. One of the most crucial aspects is the landscape at random initialization, which often influences convergenc…

Cited by 11Source ↗PDF ↗Scholar ↗
2021

Direct-Search for a Class of Stochastic Min-Max Problems

Sotirios-Konstantinos Anagnostidis, Aurelien Lucchi, Youssef Diouane
AISTATS 2021poster

Recent applications in machine learning have renewed the interest of the community in min-max optimization problems. While gradient-based optimization methods are widely used to solve such problems, there are however many scenarios where these techniques are not well-suited, or even not applicable w…

Cited by 12Source ↗PDF ↗Scholar ↗
2021

Learning Generative Models of Textured 3D Meshes From Real-World Images

Dario Pavllo, Jonas Kohler, Thomas Hofmann, Aurelien Lucchi
ICCV 2021poster

Recent advances in differentiable rendering have sparked an interest in learning generative models of textured 3D meshes from image collections. These models natively disentangle pose and appearance, enable downstream applications in computer graphics, and improve the ability of generative models to…

Cited by 56PDF ↗code ↗Scholar ↗
2021

Momentum Improves Optimization on Riemannian Manifolds

Foivos Alimisis, Antonio Orvieto, Gary Becigneul, Aurelien Lucchi
AISTATS 2021poster

We develop a new Riemannian descent algorithm that relies on momentum to improve over existing first-order methods for geodesically convex optimization. In contrast, accelerated convergence rates proved in prior work have only been shown to hold for geodesically strongly-convex objective functions.…

Cited by 53Source ↗PDF ↗code ↗Scholar ↗
2021

Neural Symbolic Regression that scales

Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, Giambattista Parascandolo
ICML 2021spotlight

Symbolic equations are at the core of scientific discovery. The task of discovering the underlying equation from a set of input-output pairs is called symbolic regression. Traditionally, symbolic regression methods use hand-designed strategies that do not improve with experience. In this paper, we i…

Cited by 232Source ↗PDF ↗code ↗Scholar ↗
2021

On the Second-order Convergence Properties of Random Search Methods

Aurelien Lucchi, Antonio Orvieto, Adamos Solomou
NeurIPS 2021poster

We study the theoretical convergence properties of random-search methods when optimizing non-convex objective functions without having access to derivatives. We prove that standard random-search methods that do not rely on second-order information converge to a second-order stationary point. However…

Cited by 9Source ↗PDF ↗code ↗Scholar ↗
2021

Scalable Graph Networks for Particle Simulations

Karolis Martinkus, Aurelien Lucchi, Nathanaël Perraudin
AAAI 2021technical

Learning system dynamics directly from observations is a promising direction in machine learning due to its potential to significantly enhance our ability to understand physical systems. However, the dynamics of many real-world systems are challenging to learn due to the presence of nonlinear potent…

Cited by 10Source ↗PDF ↗code ↗Scholar ↗
2020

A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization

Foivos Alimisis, Antonio Orvieto, Gary Becigneul, Aurelien Lucchi
AISTATS 2020poster

We propose a novel second-order ODE as the continuous-time limit of a Riemannian accelerated gradient-based method on a manifold with curvature bounded from below. This ODE can be seen as a generalization of the ODE derived for Euclidean spaces, and can also serve as an analysis tool. We analyze th…

Cited by 69Source ↗PDF ↗code ↗Scholar ↗
2020

Batch normalization provably avoids ranks collapse for randomly initialised deep networks

Hadi Daneshmand, Jonas Kohler, Francis R. Bach, Thomas Hofmann, Aurelien Lucchi
NeurIPS 2020poster

Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization are used. We here investigate this phenomenon by revisiting the connection between random initialization in deep network…

Cited by 73Source ↗PDF ↗Scholar ↗
2020

Controlling Style and Semantics in Weakly-Supervised Image Generation

Dario Pavllo, Aurelien Lucchi, Thomas Hofmann
ECCV 2020poster

We propose a weakly-supervised approach for conditional image generation of complex scenes where a user has fine control over objects appearing in the scene. We exploit sparse semantic maps to control object shapes and classes, as well as textual descriptions or attributes to control both local and…

Cited by 44Source ↗PDF ↗code ↗Scholar ↗
2020

Convolutional Generation of Textured 3D Meshes

Dario Pavllo, Graham Spinks, Thomas Hofmann, Marie-Francine Moens, Aurelien Lucchi
NeurIPS 2020oral

While recent generative models for 2D images achieve impressive visual results, they clearly lack the ability to perform 3D reasoning. This heavily restricts the degree of control over generated objects as well as the possible applications of such models. In this work, we bridge this gap by leveragi…

Cited by 72Source ↗PDF ↗code ↗Scholar ↗
2019

A Domain Agnostic Measure for Monitoring and Evaluating GANs

Paulina Grnarova, Kfir Y. Levy, Aurelien Lucchi, Nathanael Perraudin, Ian Goodfellow, Thomas Hofmann +1
NeurIPS 2019poster

Generative Adversarial Networks (GANs) have shown remarkable results in modeling complex distributions, but their evaluation remains an unsettled issue. Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training.…

Cited by 48Source ↗PDF ↗code ↗Scholar ↗
2019

Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, Klaus Neymeyr
AISTATS 2019poster

Normalization techniques such as Batch Normalization have been applied very successfully for training deep neural networks. Yet, despite its apparent empirical benefits, the reasons behind the success of Batch Normalization are mostly hypothetical. We here aim to provide a more thorough theoretical…

Cited by 112Source ↗PDF ↗Scholar ↗
2019

Local Saddle Point Optimization: A Curvature Exploitation Approach

Leonard Adolphs, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann
AISTATS 2019poster

Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that…

Cited by 142Source ↗PDF ↗Scholar ↗
2018

A Distributed Second-Order Algorithm You Can Trust

Celestine Duenner, Aurelien Lucchi, Matilde Gargiani, An Bian, Thomas Hofmann, Martin Jaggi
ICML 2018oral

Due to the rapid growth of data and computational resources, distributed optimization has become an active research area in recent years. While first-order methods seem to dominate the field, second-order methods are nevertheless attractive as they potentially require fewer communication rounds to c…

Cited by 36Source ↗PDF ↗code ↗Scholar ↗
2018

An Online Learning Approach to Generative Adversarial Networks

Paulina Grnarova, Kfir Y Levy, Aurelien Lucchi, Thomas Hofmann, Andreas Krause
ICLR 2018poster

We consider the problem of training generative models with a Generative Adversarial Network (GAN). Although GANs can accurately model complex distributions, they are known to be difficult to train due to instabilities caused by a difficult minimax optimization problem. In this paper, we view the pro…

Cited by 92Source ↗PDF ↗Scholar ↗
2017

Stabilizing Training of Generative Adversarial Networks through Regularization

Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, Thomas Hofmann
NeurIPS 2017poster

Deep generative models based on Generative Adversarial Networks (GANs) have demonstrated impressive sample quality but in order to work they require a careful choice of architecture, parameter initialization, and selection of hyper-parameters. This fragility is in part due to a dimensional mismatch…

Cited by 554Source ↗PDF ↗code ↗Scholar ↗
2016

Adaptive Newton Method for Empirical Risk Minimization to Statistical Accuracy

Aryan Mokhtari, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Alejandro Ribeiro
NeurIPS 2016poster

We consider empirical risk minimization for large-scale datasets. We introduce Ada Newton as an adaptive algorithm that uses Newton's method with adaptive sample sizes. The main idea of Ada Newton is to increase the size of the training set by a factor larger than one in a way that the minimization…

Cited by 47Source ↗PDF ↗Scholar ↗
2015

Variance Reduced Stochastic Gradient Descent with Neighbors

Thomas Hofmann, Aurelien Lucchi, Simon Lacoste-Julien, Brian McWilliams
NeurIPS 2015poster

Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet it is also known to be slow relative to steepest descent. Recently, variance reduction techniques such as SVRG and SAGA have been proposed to overcome this weakness. With asymptotically vanishing variance, a constant step siz…

Cited by 183Source ↗PDF ↗Scholar ↗