Rustem Islamov — accepted AI-conference papers

2026

Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective

Enea Monzio Compagnoni, Alessandro Stanghellini, Rustem Islamov, Aurelien Lucchi, Anastasia Koloskova

ICLR 2026poster

Differential Privacy (DP) is becoming central to large-scale training as privacy regulations tighten. We revisit how DP noise interacts with *adaptivity* in optimization through the lens of *stochastic differential equations*, providing the first SDE-based analysis of private optimizers. Focusing on…

Cited by 0 Source ↗code ↗Scholar ↗

2026

Non-Euclidean Gradient Descent Operates at the Edge of Stability

Rustem Islamov, Michael Crawshaw, Jeremy Cohen, Robert Gower

ICML 2026oral

The Edge of Stability (EoS) is a phenomenon where the sharpness (largest eigenvalue) of the Hessian converges to $2/\eta$ during training with gradient descent (GD) with a step-size $\eta$. Despite violating classical smoothness assumptions, EoS has been widely observed in deep learning, but its the…

Cited by 0 Source ↗Scholar ↗

2026

On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach

Enea Monzio Compagnoni, Rustem Islamov, Frank Proske, Aurelien Lucchi, Antonio Orvieto, Eduard Gorbunov

ICML 2026poster

Distributed stochastic optimization intertwines (i) stochastic gradient noise, (ii) communication compression, and (iii) adaptive/normalized updates. While each factor has been studied in isolation, their joint effect under realistic assumptions remains poorly understood. In this work, we develop a …

Cited by 0 Source ↗Scholar ↗

2026

On the Role of Batch Size in Stochastic Conditional Gradient Methods

Rustem Islamov, Roman Machacek, Aurelien Lucchi, Antonio Silveti-Falls, Eduard Gorbunov, Volkan Cevher

ICML 2026poster

We study the role of batch size in stochastic conditional gradient methods under a $\mu$-Kurdyka–Łojasiewicz ($\mu$-KL) condition. Focusing on momentum-based stochastic Frank–Wolfe–type conditional gradient algorithms (e.g., Scion), we derive a new analysis that explicitly captures the interaction b…

Cited by 0 Source ↗Scholar ↗

2026

Why Do We Need Warm-up? A Theoretical Perspective

Foivos Alimisis, Rustem Islamov, Aurelien Lucchi

ICML 2026poster

Learning rate warm-up -- increasing the learning rate at the beginning of training -- has become a ubiquitous heuristic in modern deep learning, yet its theoretical foundations remain poorly understood. In this work, we provide a principled explanation for why warm-up improves training. We rely on a…

Cited by 0 Source ↗Scholar ↗

2025

Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise

Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, Aurelien Lucchi

ICLR 2025poster

Despite the vast empirical evidence supporting the efficacy of adaptive optimization methods in deep learning, their theoretical understanding is far from complete. This work introduces novel SDEs for commonly used adaptive optimizers: SignSGD, RMSprop(W), and Adam(W). These SDEs offer a quantitativ…

Cited by 2 Source ↗PDF ↗Scholar ↗

2025

Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size

Rustem Islamov, Niccolò Ajroldi, Antonio Orvieto, Aurelien Lucchi

NeurIPS 2025poster

Modern optimization algorithms that incorporate momentum and adaptive step-size offer improved performance in numerous challenging deep learning tasks. However, their effectiveness is often highly sensitive to the choice of hyperparameters, especially the step-size. Tuning these parameters is often…

Cited by 0 Source ↗Scholar ↗

2025

Safe-EF: Error Feedback for Non-smooth Constrained Optimization

Rustem Islamov, Yarden As, Ilyas Fatkhullin

ICML 2025poster

Federated learning faces severe communication bottlenecks due to the high dimensionality of model updates. Communication compression with contractive compressors (e.g., Top-$K$) is often preferable in practice but can degrade performance without proper handling. Error feedback (EF) mitigates such is…

Cited by 0 Source ↗PDF ↗code ↗Scholar ↗

2025

Towards Faster Decentralized Stochastic Optimization with Communication Compression

Rustem Islamov, Yuan Gao, Sebastian U Stich

ICLR 2025poster

Communication efficiency has garnered significant attention as it is considered the main bottleneck for large-scale decentralized Machine Learning applications in distributed and federated settings. In this regime, clients are restricted to transmitting small amounts of compressed information to the…

Cited by 0 Source ↗PDF ↗code ↗Scholar ↗

2025

Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs

Enea Monzio Compagnoni, Rustem Islamov, Frank Norbert Proske, Aurelien Lucchi

AISTATS 2025oral

Distributed methods are essential for handling machine learning pipelines comprising large-scale models and datasets. However, their benefits often come at the cost of increased communication overhead between the central server and agents, which can become the main bottleneck, making training costly…

Cited by 0 Source ↗Scholar ↗

2024

AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithms

Rustem Islamov, Mher Safaryan, Dan Alistarh

AISTATS 2024poster

We analyze asynchronous-type algorithms for distributed SGD in the heterogeneous setting, where each worker has its own computation and communication speeds, as well as data distribution. In these algorithms, workers compute possibly stale and stochastic gradients associated with their local data at…

Cited by 18 Source ↗PDF ↗Scholar ↗

2024

EControl: Fast Distributed Optimization with Compression and Error Control

Yuan Gao, Rustem Islamov, Sebastian U Stich

ICLR 2024poster

Modern distributed training relies heavily on communication compression to reduce the communication overhead. In this work, we study algorithms employing a popular class of contractive compressors in order to reduce communication overhead. However, the naive implementation often leads to unstable co…

Cited by 10 Source ↗PDF ↗Scholar ↗

2024

Loss Landscape Characterization of Neural Networks without Over-Parametrization

Rustem Islamov, Niccolò Ajroldi, Antonio Orvieto, Aurelien Lucchi

NeurIPS 2024poster

Modern machine learning heavily depends on the effectiveness of optimization techniques. While deep learning models have achieved remarkable empirical results in training, their theoretical underpinnings remain somewhat elusive. Ensuring the convergence of optimization methods requires imposing spec…

Cited by 6 Source ↗PDF ↗Scholar ↗

2022

Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning

Xun Qian, Rustem Islamov, Mher Safaryan, Peter Richtarik

AISTATS 2022poster

Recent advances in distributed optimization have shown that Newton-type methods with proper communication compression mechanisms can guarantee fast local rates and low communication cost compared to first order methods. We discover that the communication cost of these methods can be further reduced,…

Cited by 28 Source ↗PDF ↗Scholar ↗

2022

FedNL: Making Newton-Type Methods Applicable to Federated Learning

Mher Safaryan, Rustem Islamov, Xun Qian, Peter Richtarik

ICML 2022spotlight

Inspired by recent work of Islamov et al (2021), we propose a family of Federated Newton Learn (\algname{FedNL}) methods, which we believe is a marked step in the direction of making second-order methods applicable to FL. In contrast to the aforementioned work, \algname{FedNL} employs a different He…

Cited by 96 Source ↗PDF ↗Scholar ↗

2021

Distributed Second Order Methods with Fast Rates and Compressed Communication

Rustem Islamov, Xun Qian, Peter Richtarik

ICML 2021spotlight

We develop several new communication-efficient second-order methods for distributed optimization. Our first method, NEWTON-STAR, is a variant of Newton’s method from which it inherits its fast local quadratic rate. However, unlike Newton’s method, NEWTON-STAR enjoys the same per iteration communicat…

Cited by 67 Source ↗PDF ↗Scholar ↗