AIConfPaper
← Search

Alexander Long

9 accepted papers

2026

Factored Gossip DiLoCo: Reducing Blocking Communication within DiLoCo

Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe, Hadi Mohaghegh Dolatabadi, Shamane Siriwardhana, Gil Avraham +4
ICML 2026poster

To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently, its outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures. We relax exact sync…

Cited by 0Source ↗Scholar ↗
2026

Taming Curvature: Architecture Warm-up for Stable Transformer Training

Sameera Ramasinghe, Thalaiyasingam Ajanthan, Hadi Mohaghegh Dolatabadi, Chamin P Hewa Koneputugodage, Gil Avraham, Violetta Shevchenko +3
ICLR 2026poster

Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) theory provides a powerful tool to understand and control the stability of optimization methods via the (preconditioned)…

Cited by 0Source ↗Scholar ↗
2025

Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training

Sameera Ramasinghe, Thalaiyasingam Ajanthan, Hadi Mohaghegh Dolatabadi, Gil Avraham, Violetta Shevchenko, Yan Zuo +2
NeurIPS 2025poster

Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute attention block by block which incurs significant communication ove…

Cited by 0Source ↗Scholar ↗
2025

Nesterov Method for Asynchronous Pipeline Parallel Optimization

Thalaiyasingam Ajanthan, Sameera Ramasinghe, Yan Zuo, Gil Avraham, Alexander Long
ICML 2025poster

Pipeline Parallelism (PP) enables large neural network training on small, interconnected devices by splitting the model into multiple stages. To maximize pipeline utilization, asynchronous optimization is appealing as it offers 100% pipeline utilization by construction. However, it is inherently cha…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham, Yan Zuo, Alexander Long
NeurIPS 2025poster

Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-pa…

Cited by 0Source ↗Scholar ↗
2025

Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization

Alexander Long, Chamin P Hewa Koneputugodage, Thalaiyasingam Ajanthan, Yan Zuo, Gil Avraham, Violetta Shevchenko +2
NeurIPS 2025poster

We consider a decentralized setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model. In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to a…

Cited by 0Source ↗Scholar ↗
2024

A sampling theory perspective on activations for implicit neural representations

Hemanth Saratchandran, Sameera Ramasinghe, Violetta Shevchenko, Alexander Long, Simon Lucey
ICML 2024poster

Implicit Neural Representations (INRs) have gained popularity for encoding signals as compact, differentiable entities. While commonly using techniques like Fourier positional encodings or non-traditional activation functions (e.g., Gaussian, sinusoid, or wavelets) to capture high-frequency content,…

Cited by 6Source ↗PDF ↗Scholar ↗
2022

Fast and Data Efficient Reinforcement Learning from Pixels via Non-parametric Value Approximation

Alexander Long, Alan Blair, Herke van Hoof
AAAI 2022technical

We present Nonparametric Approximation of Inter-Trace returns (NAIT), a Reinforcement Learning algorithm for discrete action, pixel-based environments that is both highly sample and computation efficient. NAIT is a lazy-learning approach with an update that is equivalent to episodic Monte-Carlo on e…

Cited by 3Source ↗PDF ↗code ↗Scholar ↗
2022

Retrieval Augmented Classification for Long-Tail Visual Recognition

Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg +3
CVPR 2022poster

We introduce Retrieval Augmented Classification (RAC), a generic approach to augmenting standard image classification pipelines with an explicit retrieval module. RAC consists of a standard base image encoder fused with a parallel retrieval branch that queries a non-parametric external memory of pre…

Cited by 127PDF ↗Scholar ↗