AIConfPaper
← Search

Surya Ganguli

49 accepted papers

2026

CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers

Haining Pan, James V Roggeveen, Erez Berg, Juan Felipe Carrasquilla Alvarez, Debanjan Chowdhury, Surya Ganguli +13
ICLR 2026poster

Large language models (LLMs) have demonstrated remarkable progress in coding and mathematical problem-solving; however, evaluation on advanced research-level problems in the hard sciences remains scarce. To fill this gap, we present \cmt, a dataset of 50 original problems covering condensed matter…

Cited by 0Source ↗Scholar ↗
2026

Causal Interpretation of Neural Network Computations with Contribution Decomposition (CODEC)

Joshua Brendan Melander, Zaki Alaoui, Shenghua Liu, Surya Ganguli, Stephen Baccus
ICLR 2026poster

Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human-interpretable concepts. Here we take a direct a…

Cited by 0Source ↗code ↗Scholar ↗
2026

Deriving Neural Scaling Laws from the Statistics of Natural Language

Francesco Cagnetta, Allan Raventos, Surya Ganguli, Matthieu Wyart
ICML 2026poster

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first su…

Cited by 0Source ↗Scholar ↗
2026

From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers

Ziming Liu, Surya Ganguli, Andreas Tolias
ICML 2026poster

Vafa et al. recently showed that a transformer fails to acquire an internal Newtonian world model when trained on synthetic planetary-motion data. How can we fix this problem? We find that inductive biases are key to learning the veridical world model: (1) **Spatial smoothness** is required for any …

Cited by 0Source ↗Scholar ↗
2025

Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks

Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B Simon, Michael R DeWeese +2
NeurIPS 2025poster

What features neural networks learn, and how, remains an open question. In this paper, we introduce Alternating Gradient Flows (AGF), an algorithmic framework that describes the dynamics of feature learning in two-layer networks trained from small initialization. Prior works have shown that gradient…

Cited by 0Source ↗Scholar ↗
2025

Features are fate: a theory of transfer learning in high-dimensional regression

Javan Tahir, Surya Ganguli, Grant M. Rotskoff
ICML 2025poster

With the emergence of large-scale pre-trained neural networks, methods to adapt such "foundation" models to data-limited downstream tasks have become a necessity. Fine-tuning, preference optimization, and transfer learning have all been successfully employed for these purposes when the target task c…

Cited by 1Source ↗PDF ↗Scholar ↗
2025

Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning

Feng Chen, Allan Raventos, Nan Cheng, Surya Ganguli, Shaul Druckmann
NeurIPS 2025poster

Recent progress in large language models (LLMs) highlights the power of scaling test-time compute to achieve strong performance on complex tasks, such as mathematical reasoning and code generation. This raises a critical question: how should model training be modified to optimize performance under a…

Cited by 0Source ↗code ↗Scholar ↗
2024

Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning

Daniel Kunin, Allan Raventos, Clémentine Carla Juliette Dominé, Feng Chen, David Klindt, Andrew M Saxe +1
NeurIPS 2024spotlight

While the impressive performance of modern neural networks is often attributed to their capacity to efficiently extract task-relevant features from data, the mechanisms underlying this *rich feature learning regime* remain elusive, with much of our theoretical understanding stemming from the opposin…

Cited by 13Source ↗PDF ↗code ↗Scholar ↗
2023

Disentanglement with Biological Constraints: A Theory of Functional Cell Types

James C. R. Whittington, Will Dorrell, Surya Ganguli, Timothy Behrens
ICLR 2023top-25%

Neurons in the brain are often finely tuned for specific task variables. Moreover, such disentangled representations are highly sought after in machine learning. Here we mathematically prove that simple biological constraints on neurons, namely nonnegativity and energy efficiency in both activity an…

Cited by 35Source ↗PDF ↗Scholar ↗
2023

Information Geometry of the Retinal Representation Manifold

Xuehao Ding, Dongsoo Lee, Joshua Brendan Melander, George Sivulka, Surya Ganguli, Stephen Baccus
NeurIPS 2023poster

The ability for the brain to discriminate among visual stimuli is constrained by their retinal representations. Previous studies of visual discriminability have been limited to either low-dimensional artificial stimuli or pure theoretical considerations without a realistic encoding model. Here we pr…

Cited by 6Source ↗PDF ↗Scholar ↗
2023

Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression

Allan Raventos, Mansheej Paul, Feng Chen, Surya Ganguli
NeurIPS 2023poster

Pretrained transformers exhibit the remarkable ability of in-context learning (ICL): they can learn tasks from just a few examples provided in the prompt without updating any weights. This raises a foundational question: can ICL solve fundamentally _new_ tasks that are very different from those seen…

Cited by 88Source ↗PDF ↗code ↗Scholar ↗
2023

Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks

Feng Chen, Daniel Kunin, Atsushi Yamamura, Surya Ganguli
NeurIPS 2023poster

In this work, we reveal a strong implicit bias of stochastic gradient descent (SGD) that drives overly expressive networks to much simpler subnetworks, thereby dramatically reducing the number of independent parameters, and improving generalization. To reveal this bias, we identify _invariant sets_,…

Cited by 28Source ↗PDF ↗code ↗Scholar ↗
2023

The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks

Daniel Kunin, Atsushi Yamamura, Chao Ma, Surya Ganguli
ICLR 2023top-25%

In this work, we explore the maximum-margin bias of quasi-homogeneous neural networks trained with gradient flow on an exponential loss and past a point of separability. We introduce the class of quasi-homogeneous models, which is expressive enough to describe nearly all neural networks with homogen…

Cited by 29Source ↗PDF ↗Scholar ↗
2023

Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?

Mansheej Paul, Feng Chen, Brett W. Larsen, Jonathan Frankle, Surya Ganguli, Gintare Karolina Dziugaite
ICLR 2023top-25%

As neural networks get larger and costlier, it is important to find sparse networks that require less compute and memory but can be trained to the same accuracy as the full network (i.e. matching). Iterative magnitude pruning (IMP) is a state of the art algorithm that can find such highly sparse mat…

Cited by 50Source ↗PDF ↗Scholar ↗
2022

Beyond neural scaling laws: beating power law scaling via data pruning

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos
NeurIPS 2022accept

Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus…

Cited by 445Source ↗PDF ↗code ↗Scholar ↗
2022

How many degrees of freedom do we need to train deep networks: a loss landscape perspective

Brett W Larsen, Stanislav Fort, Nic Becker, Surya Ganguli
ICLR 2022poster

A variety of recent works, spanning pruning, lottery tickets, and training within random subspaces, have shown that deep neural networks can be trained using far fewer degrees of freedom than the total number of parameters. We analyze this phenomenon for random subspaces by first examining the succe…

Cited by 27Source ↗PDF ↗code ↗Scholar ↗
2022

Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks

Mansheej Paul, Brett W Larsen, Surya Ganguli, Jonathan Frankle, Gintare Karolina Dziugaite
NeurIPS 2022accept

A striking observation about iterative magnitude pruning (IMP; Frankle et al. 2020) is that—after just a few hundred steps of dense training—the method can find a sparse sub-network that can be trained to the same accuracy as the dense network. However, the same does not hold at step 0, i.e. random…

Cited by 19Source ↗PDF ↗code ↗Scholar ↗
2022

MetaMorph: Learning Universal Controllers with Transformers

Agrim Gupta, Linxi Fan, Surya Ganguli, Li Fei-Fei
ICLR 2022poster

Multiple domains like vision, natural language, and audio are witnessing tremendous progress by leveraging Transformers for large scale pre-training followed by task specific fine tuning. In contrast, in robotics we primarily train a single robot for a single task. However, modular robot systems now…

Cited by 97Source ↗PDF ↗code ↗Scholar ↗
2021

A theory of high dimensional regression with arbitrary correlations between input features and target functions: sample complexity, multiple descent curves and a hierarchy of phase transitions

Gabriel Mel, Surya Ganguli
ICML 2021spotlight

The performance of neural networks depends on precise relationships between four distinct ingredients: the architecture, the loss function, the statistical structure of inputs, and the ground truth target function. Much theoretical work has focused on understanding the role of the first two ingredie…

Cited by 28Source ↗PDF ↗Scholar ↗
2021

Deep Learning on a Data Diet: Finding Important Examples Early in Training

Mansheej Paul, Surya Ganguli, Gintare Karolina Dziugaite
NeurIPS 2021poster

Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. It is therefore natural to ask: how much of the data is superfluous, which examples are important for generalization, and how do we find them? In this work, we make t…

Cited by 502Source ↗PDF ↗code ↗Scholar ↗
2021

Explaining heterogeneity in medial entorhinal cortex with task-driven neural networks

Aran Nayebi, Alexander Attinger, Malcolm G. Campbell, Kiah Hardcastle, Isabel I. C. Low, Caitlin S. Mallory +6
NeurIPS 2021spotlight

Medial entorhinal cortex (MEC) supports a wide range of navigational and memory related behaviors. Well-known experimental results have revealed specialized cell types in MEC --- e.g. grid, border, and head-direction cells --- whose highly stereotypical response profiles are suggestive of the role t…

Cited by 45Source ↗PDF ↗code ↗Scholar ↗
2021

Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics

Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel LK Yamins, Hidenori Tanaka
ICLR 2021poster

Understanding the dynamics of neural network parameters during training is one of the key challenges in building a theoretical foundation for deep learning. A central obstacle is that the motion of a network in high-dimensional parameter space undergoes discrete finite steps along complex stochastic…

Cited by 86Source ↗PDF ↗code ↗Scholar ↗
2021

Understanding self-supervised learning dynamics without contrastive pairs

Yuandong Tian, Xinlei Chen, Surya Ganguli
ICML 2021oral

While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs), recent \emph{non-contrastive} SSL (e.g., BYOL and SimS…

Cited by 349Source ↗PDF ↗code ↗Scholar ↗
2020

Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, Surya Ganguli
NeurIPS 2020poster

In suitably initialized wide networks, small learning rates transform deep neural networks (DNNs) into neural tangent kernel (NTK) machines, whose training dynamics is well-approximated by a linear weight expansion of the network at initialization. Standard training, however, diverges from its line…

Cited by 227Source ↗PDF ↗Scholar ↗
2020

Identifying Learning Rules From Neural Network Observables

Aran Nayebi, Sanjana Srivastava, Surya Ganguli, Daniel L Yamins
NeurIPS 2020spotlight

The brain modifies its synaptic strengths during learning in order to better adapt to its environment. However, the underlying plasticity rules that govern learning are unknown. Many proposals have been suggested, including Hebbian mechanisms, explicit error backpropagation, and a variety of alterna…

Cited by 36Source ↗PDF ↗code ↗Scholar ↗
2020

Predictive coding in balanced neural networks with noise, chaos and delays

Jonathan Kadmon, Jonathan Timcheck, Surya Ganguli
NeurIPS 2020poster

Biological neural networks face a formidable task: performing reliable computations in the face of intrinsic stochasticity in individual neurons, imprecisely specified synaptic connectivity, and nonnegligible delays in synaptic transmission. A common approach to combatting such biological heterogen…

Cited by 43Source ↗PDF ↗Scholar ↗
2020

Pruning neural networks without any data by iteratively conserving synaptic flow

Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, Surya Ganguli
NeurIPS 2020poster

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery ticket…

Cited by 805Source ↗PDF ↗code ↗Scholar ↗
2020

Two Routes to Scalable Credit Assignment without Weight Symmetry

Daniel Kunin, Aran Nayebi, Javier Sagastuy-Brena, Surya Ganguli, Jonathan Bloom, Daniel Yamins
ICML 2020poster

The neural plausibility of backpropagation has long been disputed, primarily for its use of non-local weight transport — the biologically dubious requirement that one neuron instantaneously measure the synaptic weights of another. Until recently, attempts to create local learning rules that avoid we…

Cited by 45Source ↗PDF ↗code ↗Scholar ↗
2019

A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs

Jack Lindsey, Samuel A. Ocko, Surya Ganguli, Stephane Deny
ICLR 2019oral

The vertebrate visual system is hierarchically organized to process visual information in successive stages. Neural representations vary drastically across the first stages of visual processing: at the output of the retina, ganglion cell receptive fields (RFs) exhibit a clear antagonistic center-sur…

Cited by 104Source ↗PDF ↗code ↗Scholar ↗
2019

A unified theory for the origin of grid cells through the lens of pattern formation

Ben Sorscher, Gabriel Mel, Surya Ganguli, Samuel Ocko
NeurIPS 2019spotlight

Grid cells in the brain fire in strikingly regular hexagonal patterns across space. There are currently two seemingly unrelated frameworks for understanding these patterns. Mechanistic models account for hexagonal firing fields as the result of pattern-forming dynamics in a recurrent neural network…

Cited by 86Source ↗PDF ↗code ↗Scholar ↗
2019

An analytic theory of generalization dynamics and transfer learning in deep linear networks

Andrew K. Lampinen, Surya Ganguli
ICLR 2019poster

Much attention has been devoted recently to the generalization puzzle in deep learning: large, deep networks can generalize well, but existing theories bounding generalization error are exceedingly loose, and thus cannot explain this striking performance. Furthermore, a major hope is that knowledge…

Cited by 136Source ↗PDF ↗Scholar ↗
2019

From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction

Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, Surya Ganguli
NeurIPS 2019poster

Recently, deep feedforward neural networks have achieved considerable success in modeling biological sensory processing, in terms of reproducing the input-output map of sensory neurons. However, such models raise profound questions about the very nature of explanation in neuroscience. Are we simply…

Cited by 87Source ↗PDF ↗code ↗Scholar ↗
2019

Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics

Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, David Sussillo
NeurIPS 2019poster

Recurrent neural networks (RNNs) are a widely used tool for modeling sequential data, yet they are often treated as inscrutable black boxes. Given a trained recurrent network, we would like to reverse engineer it--to obtain a quantitative, interpretable description of how it solves a particular task…

Cited by 104Source ↗PDF ↗Scholar ↗
2019

Universality and individuality in neural dynamics across large populations of recurrent networks

Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, David Sussillo
NeurIPS 2019spotlight

Many recent studies have employed task-based modeling with recurrent neural networks (RNNs) to infer the computational function of different brain regions. These models are often assessed by quantitatively comparing the low-dimensional neural dynamics of the model and the brain, for example using ca…

Cited by 174Source ↗PDF ↗Scholar ↗
2018

Task-Driven Convolutional Recurrent Models of the Visual System

Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo +2
NeurIPS 2018poster

Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems ha…

Cited by 195Source ↗PDF ↗code ↗Scholar ↗
2018

The emergence of multiple retinal cell types through efficient coding of natural movies

Samuel Ocko, Jack Lindsey, Surya Ganguli, Stephane Deny
NeurIPS 2018poster

One of the most striking aspects of early visual processing in the retina is the immediate parcellation of visual information into multiple parallel pathways, formed by different retinal ganglion cell types each tiling the entire visual field. Existing theories of efficient coding have been unable t…

Cited by 56Source ↗PDF ↗code ↗Scholar ↗
2018

The emergence of spectral universality in deep networks

Jeffrey Pennington, Samuel Schoenholz, Surya Ganguli
AISTATS 2018poster

Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network’s input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical unde…

Cited by 0Source ↗PDF ↗Scholar ↗
2017

On the Expressive Power of Deep Neural Networks

Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl-Dickstein
ICML 2017poster

We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion…

Cited by 1057Source ↗PDF ↗Scholar ↗
2017

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

Jeffrey Pennington, Samuel Schoenholz, Surya Ganguli
NeurIPS 2017poster

It is well known that weight initialization in deep networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is O(1) is essential for avoiding exponentially vanishing or exploding gradients. Moreover, in deep li…

Cited by 335Source ↗PDF ↗Scholar ↗
2017

Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net

Anirudh Goyal ALIAS PARTH GOYAL, Nan Rosemary Ke, Surya Ganguli, Yoshua Bengio
NeurIPS 2017poster

We propose a novel method to {\it directly} learn a stochastic transition operator whose repeated application provides generated samples. Traditional undirected graphical models approach this problem indirectly by learning a Markov chain model whose stationary distribution obeys detailed balance wit…

Cited by 60Source ↗PDF ↗code ↗Scholar ↗
2016

Deep Learning Models of the Retinal Response to Natural Scenes

Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, Stephen Baccus
NeurIPS 2016poster

A central challenge in sensory neuroscience is to understand neural computations and circuit mechanisms that underlie the encoding of ethologically relevant, natural stimuli. In multilayered neural circuits, nonlinear processes such as synaptic transmission and spiking dynamics present a significant…

Cited by 318Source ↗PDF ↗Scholar ↗
2016

Exponential expressivity in deep neural networks through transient chaos

Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, Surya Ganguli
NeurIPS 2016poster

We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in deep neural networks with random weights. Our results reveal a phase transition in the expressivity of random deep networks, with networks in the chaotic phase computing n…

Cited by 717Source ↗PDF ↗code ↗Scholar ↗
2015

Deep Knowledge Tracing

Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas +1
NeurIPS 2015poster

Knowledge tracing, where a machine models the knowledge of a student as they interact with coursework, is an established and significantly unsolved problem in computer supported education.In this paper we explore the benefit of using recurrent neural networks to model student learning.This family of…

Cited by 1838Source ↗PDF ↗code ↗Scholar ↗
2015

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, Surya Ganguli
ICML 2015poster

A central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable. Here, we develop an approach that simultaneously achieves…

Cited by 8219Source ↗PDF ↗code ↗Scholar ↗