AIConfPaper
← Search

Arna Ghosh

8 accepted papers

2025

Tracing the Representation Geometry of Language Models from Pretraining to Post-training

Melody Zixuan Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Kumar Teru, Adam Santoro, Guillaume Lajoie +1
NeurIPS 2025poster

Standard training metrics like loss fail to explain the emergence of complex capabilities in large language models. We take a spectral approach to investigate the geometry of learned representations across pretraining and post-training, measuring effective rank (RankMe) and eigenspectrum decay (αReQ…

Cited by 0Source ↗Scholar ↗
2024

Harnessing small projectors and multiple views for efficient vision pretraining

Arna Ghosh, Kumar Krishna Agrawal, Shagun Sodhani, Adam Oberman, Blake Aaron Richards
NeurIPS 2024poster

Recent progress in self-supervised (SSL) visual representation learning has led to the development of several different proposed frameworks that rely on augmentations of images but use different loss functions. However, there are few theoretically grounded principles to guide practice, so practical…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

Learning Successor Features the Simple Way

Raymond Chua, Arna Ghosh, Christos Kaplanis, Blake Aaron Richards, Doina Precup
NeurIPS 2024poster

In Deep Reinforcement Learning (RL), it is a challenge to learn representations that do not exhibit catastrophic forgetting or interference in non-stationary environments. Successor Features (SFs) offer a potential solution to this challenge. However, canonical techniques for learning SFs from pixel…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

Synaptic Weight Distributions Depend on the Geometry of Plasticity

Roman Pogodin, Jonathan Cornford, Arna Ghosh, Gauthier Gidel, Guillaume Lajoie, Blake Aaron Richards
ICLR 2024spotlight

A growing literature in computational neuroscience leverages gradient descent and learning algorithms that approximate it to study synaptic plasticity in the brain. However, the vast majority of this work ignores a critical underlying assumption: the choice of distance for synaptic changes - i.e. th…

Cited by 6Source ↗PDF ↗code ↗Scholar ↗
2023

How gradient estimator variance and bias impact learning in neural networks

Arna Ghosh, Yuhan Helena Liu, Guillaume Lajoie, Konrad Kording, Blake Aaron Richards
ICLR 2023poster

There is growing interest in understanding how real brains may approximate gradients and how gradients can be used to train neuromorphic chips. However, neither real brains nor neuromorphic chips can perfectly follow the loss gradient, so parameter updates would necessarily use gradient estimators t…

Cited by 9Source ↗PDF ↗Scholar ↗
2023

Learning better with Dale’s Law: A Spectral Perspective

Pingsheng Li, Jonathan Cornford, Arna Ghosh, Blake Aaron Richards
NeurIPS 2023poster

Most recurrent neural networks (RNNs) do not include a fundamental constraint of real neural circuits: Dale's Law, which implies that neurons must be excitatory (E) or inhibitory (I). Dale's Law is generally absent from RNNs because simply partitioning a standard network's units into E and I populat…

Cited by 18Source ↗PDF ↗code ↗Scholar ↗
2022

$\alpha$-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay

Kumar Krishna Agrawal, Arnab Kumar Mondal, Arna Ghosh, Blake Aaron Richards
NeurIPS 2022accept

Self-Supervised Learning (SSL) with large-scale unlabelled datasets enables learning useful representations for multiple downstream tasks. However, assessing the quality of such representations efficiently poses nontrivial challenges. Existing approaches train linear probes (with frozen features) to…

Cited by 35Source ↗PDF ↗Scholar ↗
2022

Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules

Yuhan Helena Liu, Arna Ghosh, Blake Aaron Richards, Eric Todd SheaBrown, Guillaume Lajoie
NeurIPS 2022accept

To unveil how the brain learns, ongoing work seeks biologically-plausible approximations of gradient descent algorithms for training recurrent neural networks (RNNs). Yet, beyond task accuracy, it is unclear if such learning rules converge to solutions that exhibit different levels of generalizatio…

Cited by 11Source ↗PDF ↗code ↗Scholar ↗