AIConfPaper
← Search

Suriya Gunasekar

18 accepted papers

2024

Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models

Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi +2
ICLR 2024poster

We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text. We propose modeling factual queries as constraint satisfaction problems and use this framework to investigate how the LLM interacts internally with factual constraints.…

Cited by 39Source ↗PDF ↗code ↗Scholar ↗
2024

KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval

Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Ghosh Peshawaria +2
ICLR 2024poster

We study the ability of state-of-the art models to answer constraint satisfaction queries for information retrieval (e.g., “a list of ice cream shops in San Diego”). In the past, such queries were considered as tasks that could only be solved via web-search or knowledge bases. More recently, large l…

Cited by 10Source ↗PDF ↗Scholar ↗
2023

(S)GD over Diagonal Linear Networks: Implicit bias, Large Stepsizes and Edge of Stability

Mathieu Even, Scott Pesme, Suriya Gunasekar, Nicolas Flammarion
NeurIPS 2023poster

In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over $2$-layer diagonal linear networks. We prove the convergence of GD and SGD with macroscopic stepsizes in an overparametrise…

Cited by 16Source ↗PDF ↗Scholar ↗
2022

Neural-Sim: Learning to Generate Training Data with NeRF

Yunhao Ge, Harkirat Behl, Jiashu Xu, Suriya Gunasekar, Neel Joshi, Yale Song +3
ECCV 2022poster

"Traditional approaches for training a computer vision models requires collecting and labelling vast amounts of imagery under a diverse set of scene configurations and properties. This process is incredibly time-consuming, and it is challenging to ensure that the captured data distribution maps well…

Cited by 35Source ↗PDF ↗code ↗Scholar ↗
2021

Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent

Suriya Gunasekar, Blake Woodworth, Nathan Srebro
AISTATS 2021poster

We present a direct (primal only) derivation of Mirror Descent as a “partial” discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential function. We contrast this discretization to Natural Gradient Descent, which is obtained by a…

Cited by 40Source ↗PDF ↗Scholar ↗
2020

Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy

Edward Moroshko, Blake E Woodworth, Suriya Gunasekar, Jason Lee, Nati Srebro, Daniel Soudry
NeurIPS 2020spotlight

We provide a detailed asymptotic study of gradient flow trajectories and their implicit optimization bias when minimizing the exponential loss over "diagonal linear networks". This is the simplest model displaying a transition between "kernel" and non-kernel ("rich" or "active") regimes. We show ho…

Cited by 99Source ↗PDF ↗Scholar ↗
2020

Implicit Regularization and Convergence for Weight Normalization

Xiaoxia Wu, Edgar Dobriban, Tongzheng Ren, Shanshan Wu, Zhiyuan Li, Suriya Gunasekar +2
NeurIPS 2020poster

Normalization methods such as batch, weight, instance, and layer normalization are commonly used in modern machine learning. Here, we study the weight normalization (WN) method \cite{salimans2016weight} and a variant called reparametrized projected gradient descent (rPGD) for overparametrized least…

Cited by 26Source ↗PDF ↗Scholar ↗
2019

Convergence of Gradient Descent on Separable Data

Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, Daniel Soudry
AISTATS 2019poster

We provide a detailed study on the implicit bias of gradient descent when optimizing loss functions with strictly monotone tails, such as the logistic loss, over separable datasets. We look at two basic questions: (a) what are the conditions on the tail of the loss function under which gradient desc…

Cited by 186Source ↗PDF ↗Scholar ↗
2019

Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, Daniel Soudry
ICML 2019oral

With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non homogeneous models, extending previous work that focused on infinitesimal regularization on…

Cited by 86Source ↗PDF ↗Scholar ↗
2018

Characterizing Implicit Bias in Terms of Optimization Geometry

Suriya Gunasekar, Jason Lee, Daniel Soudry, Nathan Srebro
ICML 2018oral

We study the bias of generic optimization methods, including Mirror Descent, Natural Gradient Descent and Steepest Descent with respect to different potentials and norms, when optimizing underdetermined linear models or separable linear classification problems. We ask the question of whether the glo…

Cited by 522Source ↗PDF ↗Scholar ↗
2018

Implicit Bias of Gradient Descent on Linear Convolutional Networks

Suriya Gunasekar, Jason Lee, Daniel Soudry, Nati Srebro
NeurIPS 2018poster

We show that gradient descent on full-width linear convolutional networks of depth $L$ converges to a linear predictor related to the $\ell_{2/L}$ bridge penalty in the frequency domain. This is in contrast to linearly fully connected networks, where gradient descent converges to the hard margin lin…

Cited by 493Source ↗PDF ↗Scholar ↗
2018

On preserving non-discrimination when combining expert advice

Avrim Blum, Suriya Gunasekar, Thodoris Lykouris, Nati Srebro
NeurIPS 2018poster

We study the interplay between sequential decision making and avoiding discrimination against protected groups, when examples arrive online and do not follow distributional assumptions. We consider the most basic extension of classical online learning: Given a class of predictors that are individual…

Cited by 34Source ↗PDF ↗Scholar ↗
2017

Implicit Regularization in Matrix Factorization

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, Nati Srebro
NeurIPS 2017spotlight

We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix $X$ with gradient descent on a factorization of X. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, grad…

Cited by 602Source ↗PDF ↗Scholar ↗
2015

Consistent Collective Matrix Completion under Joint Low Rank Structure

Suriya Gunasekar, Makoto Yamada, Dawei Yin, Yi Chang
AISTATS 2015poster

We address the collective matrix completion problem of jointly recovering a collection of matrices with shared structure from partial (and potentially noisy) observations. To ensure well–posedness of the problem, we impose a joint low rank structure, wherein each component matrix is low rank and th…

Cited by 27Source ↗PDF ↗Scholar ↗
2015

Unified View of Matrix Completion under General Structural Constraints

Suriya Gunasekar, Arindam Banerjee, Joydeep Ghosh
NeurIPS 2015poster

Matrix completion problems have been widely studied under special low dimensional structures such as low rank or structure induced by decomposable norms. In this paper, we present a unified analysis of matrix completion under general low-dimensional structural constraints induced by {\em any} norm r…

Cited by 17Source ↗PDF ↗Scholar ↗