AIConfPaper
← Search

Inderjit S Dhillon

49 accepted papers

2026

PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation

Onkar Susladkar, Tushar Prakash, Adheesh Juvekar, Kiet A. Nguyen, Dong-Hwan Jang, Inderjit S Dhillon +1
CVPR 2026

Discrete video VAEs underpin modern text-to-video generation and video understanding systems, yet existing tokenizers typically learn visual codebooks at a single scale with limited vocabularies and shallow language supervision, leading to poor cross-modal alignment and zero-shot transfer. We introd

Cited by 0Source ↗code ↗Scholar ↗
2026

The Art of Scaling Reinforcement Learning Compute for LLMs

Fnu Devvrit, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer +3
ICLR 2026oral

Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algo…

Cited by 0Source ↗Scholar ↗
2025

Geometric Median (GM) Matching for Robust k-Subset Selection from Noisy Data

Anish Acharya, Sujay Sanghavi, Alex Dimakis, Inderjit S Dhillon
ICML 2025poster

Data pruning -- the combinatorial task of selecting a small and representative subset from a large dataset, is crucial for mitigating the enormous computational costs associated with training data-hungry modern deep learning models at scale. Since large-scale data collections are invariably noisy, d…

Cited by 0Source ↗PDF ↗Scholar ↗
2025

Large Language Models are Interpretable Learners

Ruochen Wang, Si Si, Felix Yu, Dorothea Wiesmann Rothuizen, Cho-Jui Hsieh, Inderjit S Dhillon
ICLR 2025poster

The trade-off between expressiveness and interpretability remains a core challenge when building human-centric models for classification and decision-making. While symbolic rules offer interpretability, they often lack expressiveness, whereas neural networks excel in performance but are known for be…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization

Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon +2
ICLR 2025oral

Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the updates depending on how the two LoRA factors are scaled or rotated. This deficiency leads to ineffic…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2025

Retraining with Predicted Hard Labels Provably Increases Model Accuracy

Rudrajit Das, Inderjit S Dhillon, Alessandro Epasto, Adel Javanmard, Jieming Mao, Vahab Mirrokni +2
ICML 2025poster

The performance of a model trained with noisy labels is often improved by simply *retraining* the model with its *own predicted hard labels* (i.e., $1$/$0$ labels). Yet, a detailed theoretical characterization of this phenomenon is lacking. In this paper, we theoretically analyze retraining in a lin…

Cited by 2Source ↗PDF ↗Scholar ↗
2025

Scalable In-context Ranking with Generative Models

Nilesh Gupta, Chong You, Srinadh Bhojanapalli, Sanjiv Kumar, Inderjit S Dhillon, Felix X. Yu
NeurIPS 2025poster

In-context Ranking (ICR) is an emerging paradigm for Information Retrieval (IR), which leverages contextual understanding of LLMs by directly incorporating the task description, candidate documents, and the query into the model's input prompt and tasking the LLM to identify relevant document(s). Whi…

Cited by 0Source ↗Scholar ↗
2024

Combining Axes Preconditioners through Kronecker Approximation for Deep Learning

Sai Surya Duvvuri, Fnu Devvrit, Rohan Anil, Cho-Jui Hsieh, Inderjit S Dhillon
ICLR 2024poster

Adaptive regularization based optimization methods such as full-matrix Adagrad which use gradient second-moment information hold significant potential for fast convergence in deep neural network (DNN) training, but are memory intensive and computationally demanding for large neural nets. We develop…

Cited by 6Source ↗PDF ↗Scholar ↗
2024

Dual-Encoders for Extreme Multi-label Classification

Nilesh Gupta, Fnu Devvrit, Ankit Singh Rawat, Srinadh Bhojanapalli, Prateek Jain, Inderjit S Dhillon
ICLR 2024poster

Dual-encoder (DE) models are widely used in retrieval tasks, most commonly studied on open QA benchmarks that are often characterized by multi-class and limited training data. In contrast, their performance in multi-label and data-rich retrieval settings like extreme multi-label classification (XMC)…

Cited by 15Source ↗PDF ↗code ↗Scholar ↗
2024

MatFormer: Nested Transformer for Elastic Inference

Fnu Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit S Dhillon +5
NeurIPS 2024poster

Foundation models are applied in a broad spectrum of settings with different inference constraints, from massive multi-accelerator clusters to resource-constrained standalone mobile devices. However, the substantial costs associated with training these models often limit the number of unique model s…

Cited by 11Source ↗PDF ↗Scholar ↗
2024

PRISM: A New Lens for Improved Color Understanding

Arjun Reddy Akula, Garima Pruthi, Inderjit S Dhillon, Pradyumna Narayana, Sugato Basu, Varun Jampani
EMNLP 2024industry

While image-text pre-trained models, such as CLIP, have demonstrated impressive capabilities in learning robust text and image representations, a critical area for substantial improvement remains—precise color understanding. In this paper, we address this limitation by introducing PRISM, a simple ye…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

Two-stage LLM Fine-tuning with Less Specialization and More Generalization

Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh +2
ICLR 2024poster

Pretrained large language models (LLMs) are general purpose problem solvers applicable to a diverse set of tasks with prompts. They can be further improved towards a specific task by fine-tuning on a specialized dataset. However, fine-tuning usually makes the model narrowly specialized on this datas…

Cited by 6Source ↗PDF ↗Scholar ↗
2023

A Computationally Efficient Sparsified Online Newton Method

Fnu Devvrit, Sai Surya Duvvuri, Rohan Anil, Vineet Gupta, Cho-Jui Hsieh, Inderjit S Dhillon
NeurIPS 2023poster

Second-order methods hold significant promise for enhancing the convergence of deep neural network training; however, their large memory and computational demands have limited their practicality. Thus there is a need for scalable second-order methods that can efficiently train large models. In this…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2023

Block Low-Rank Preconditioner with Shared Basis for Stochastic Optimization

Jui-Nan Yen, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh
NeurIPS 2023poster

Adaptive methods with non-diagonal preconditioning have shown state-of-the-art results on various tasks. However, their computational complexity and memory requirement makes it challenging to scale these methods to modern neural network architectures. To address this challenge, some previous works h…

Cited by 4Source ↗PDF ↗Scholar ↗
2023

Sample Efficiency of Data Augmentation Consistency Regularization

Shuo Yang, Yijun Dong, Rachel Ward, Inderjit S. Dhillon, Sujay Sanghavi, Qi Lei
AISTATS 2023poster

Data augmentation is popular in the training of large neural networks; however, currently, theoretical understanding of the discrepancy between different algorithmic choices of leveraging augmented data remains limited. In this paper, we take a step in this direction – we first present a simple and…

Cited by 25Source ↗PDF ↗Scholar ↗
2022

ELIAS: End-to-End Learning to Index and Search in Large Output Spaces

Nilesh Gupta, Patrick CHen, Hsiang-Fu Yu, Cho-Jui Hsieh, Inderjit S Dhillon
NeurIPS 2022accept

Extreme multi-label classification (XMC) is a popular framework for solving many real-world problems that require accurate prediction from a very large number of potential output choices. A popular approach for dealing with the large label space is to arrange the labels into a shallow tree-based ind…

Cited by 12Source ↗PDF ↗code ↗Scholar ↗
2022

Faster non-convex federated learning via global and local momentum

Rudrajit Das, Anish Acharya, Abolfazl Hashemi, Sujay Sanghavi, Inderjit S. Dhillon, Ufuk Topcu
UAI 2022poster

We propose \texttt{FedGLOMO}, a novel federated learning (FL) algorithm with an iteration complexity of $\mathcal{O}(\epsilon^{-1.5})$ to converge to an $\epsilon$-stationary point (i.e., $\mathbb{E}[\|\nabla f(x)\|^2] \leq \epsilon$) for smooth non-convex functions – under arbitrary client heteroge…

Cited by 106Source ↗PDF ↗Scholar ↗
2022

Linear Bandit Algorithms with Sublinear Time Complexity

Shuo Yang, Tongzheng Ren, Sanjay Shakkottai, Eric Price, Inderjit S. Dhillon, Sujay Sanghavi
ICML 2022spotlight

We propose two linear bandits algorithms with per-step complexity sublinear in the number of arms $K$. The algorithms are designed for applications where the arm set is extremely large and slowly changing. Our key realization is that choosing an arm reduces to a maximum inner product search (MIPS) p…

Cited by 18Source ↗PDF ↗Scholar ↗
2022

Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction

Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic +1
ICLR 2022poster

Learning on graphs has attracted significant attention in the learning community due to numerous real-world applications. In particular, graph neural networks (GNNs), which take \emph{numerical} node features and graph structure as inputs, have been shown to achieve state-of-the-art performance on v…

Cited by 157Source ↗PDF ↗code ↗Scholar ↗
2022

Robust Training in High Dimensions via Block Coordinate Geometric Median Descent

Anish Acharya, Abolfazl Hashemi, Prateek Jain, Sujay Sanghavi, Inderjit S. Dhillon, Ufuk Topcu
AISTATS 2022poster

Geometric median (GM) is a classical method in statistics for achieving robust estimation of the uncorrupted data; under gross corruption, it achieves the optimal breakdown point of 1/2. However, its computational complexity makes it infeasible for robustifying stochastic gradient descent (SGD) in h…

Cited by 38Source ↗PDF ↗code ↗Scholar ↗
2021

DRONE: Data-aware Low-rank Compression for Large NLP Models

Patrick CHen, Hsiang-Fu Yu, Inderjit S Dhillon, Cho-Jui Hsieh
NeurIPS 2021poster

The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings efficiency challenges, including inference speed and model size when deploying models on mobile devices. Specifically, m…

Cited by 63Source ↗PDF ↗Scholar ↗
2021

Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification

Jiong Zhang, Wei-Cheng Chang, Hsiang-Fu Yu, Inderjit S Dhillon
NeurIPS 2021poster

Extreme multi-label text classification~(XMC) seeks to find relevant labels from an extreme large label collection for a given text input. Many real-world applications can be formulated as XMC problems, such as recommendation systems, document tagging and semantic search. Recently, transformer based…

Cited by 125Source ↗PDF ↗code ↗Scholar ↗
2021

Label Disentanglement in Partition-based Extreme Multilabel Classification

Xuanqing Liu, Wei-Cheng Chang, Hsiang-Fu Yu, Cho-Jui Hsieh, Inderjit S Dhillon
NeurIPS 2021poster

Partition-based methods are increasingly-used in extreme multi-label classification (XMC) problems due to their scalability to large output spaces (e.g., millions or more). However, existing methods partition the large label space into mutually exclusive clusters, which is sub-optimal when labels ha…

Cited by 18Source ↗PDF ↗Scholar ↗
2021

Top-k eXtreme Contextual Bandits with Arm Hierarchy

Rajat Sen, Alexander Rakhlin, Lexing Ying, Rahul Kidambi, Dean Foster, Daniel N Hill +1
ICML 2021spotlight

Motivated by modern applications, such as online advertisement and recommender systems, we study the top-$k$ extreme contextual bandits problem, where the total number of arms can be enormous, and the learner is allowed to select $k$ arms and observe all or some of the rewards for the chosen arms. W…

Cited by 31Source ↗PDF ↗Scholar ↗
2019

AutoAssist: A Framework to Accelerate Training of Deep Neural Networks

Jiong Zhang, Hsiang-Fu Yu, Inderjit S Dhillon
NeurIPS 2019poster

Deep neural networks have yielded superior performance in many contemporary applications. However, the gradient computation in a deep model with millions of instances leads to a lengthy training process even with modern GPU/TPU hardware acceleration. In this paper, we propose AutoAssist, a simple f…

Cited by 39Source ↗PDF ↗code ↗Scholar ↗
2019

Inverting Deep Generative models, One layer at a time

Qi Lei, Ajil Jalal, Inderjit S Dhillon, Alexandros G Dimakis
NeurIPS 2019poster

We study the problem of inverting a deep generative model with ReLU activations. Inversion corresponds to finding a latent code vector that explains observed measurements as much as possible. In most prior works this is performed by attempting to solve a non-convex optimization problem involving t…

Cited by 66Source ↗PDF ↗code ↗Scholar ↗
2019

Parallel Asynchronous Stochastic Coordinate Descent with Auxiliary Variables

Hsiang-Fu Yu, Cho-Jui Hsieh, Inderjit S. Dhillon
AISTATS 2019poster

The key to the recent success of coordinate descent (CD) in many applications is to maintain a set of auxiliary variables to facilitate efficient single variable updates. For example, the vector of residual/primal variables has to be maintained when CD is applied for Lasso/linear SVM, respectively.…

Cited by 3Source ↗PDF ↗code ↗Scholar ↗
2019

Primal-Dual Block Generalized Frank-Wolfe

Qi Lei, JIACHENG ZHUO, Constantine Caramanis, Inderjit S Dhillon, Alexandros G Dimakis
NeurIPS 2019poster

We propose a generalized variant of Frank-Wolfe algorithm for solving a class of sparse/low-rank optimization problems. Our formulation includes Elastic Net, regularized SVMs and phase retrieval as special cases. The proposed Primal-Dual Block Generalized Frank-Wolfe algorithm reduces the per-iterat…

Cited by 16Source ↗PDF ↗code ↗Scholar ↗
2019

The Limitations of Adversarial Training and the Blind-Spot Attack

Huan Zhang*, Hongge Chen*, Zhao Song, Duane Boning, Inderjit S. Dhillon, Cho-Jui Hsieh
ICLR 2019poster

The adversarial training procedure proposed by Madry et al. (2018) is one of the most effective methods to defend against adversarial examples in deep neural net- works (DNNs). In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiv…

Cited by 194Source ↗PDF ↗Scholar ↗
2019

Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting

Rajat Sen, Hsiang-Fu Yu, Inderjit S Dhillon
NeurIPS 2019poster

Forecasting high-dimensional time series plays a crucial role in many applications such as demand forecasting and financial predictions. Modern datasets can have millions of correlated time-series that evolve together, i.e they are extremely high dimensional (one dimension for each individual time-s…

Cited by 524Source ↗PDF ↗code ↗Scholar ↗
2017

A Greedy Approach for Budgeted Maximum Inner Product Search

Hsiang-Fu Yu, Cho-Jui Hsieh, Qi Lei, Inderjit S Dhillon
NeurIPS 2017poster

Maximum Inner Product Search (MIPS) is an important task in many machine learning applications such as the prediction phase of low-rank matrix factorization models and deep learning models. Recently, there has been substantial research on how to perform MIPS in sub-linear time, but most of the exist…

Cited by 65Source ↗PDF ↗Scholar ↗
2017

Doubly Greedy Primal-Dual Coordinate Descent for Sparse Empirical Risk Minimization

Qi Lei, Ian En-Hsu Yen, Chao-yuan Wu, Inderjit S. Dhillon, Pradeep Ravikumar
ICML 2017poster

We consider the popular problem of sparse empirical risk minimization with linear predictors and a large number of both features and observations. With a convex-concave saddle point objective reformulation, we propose a Doubly Greedy Primal-Dual Coordinate Descent algorithm that is able to exploit s…

Cited by 24Source ↗PDF ↗Scholar ↗
2017

Gradient Boosted Decision Trees for High Dimensional Sparse Output

Si Si, Huan Zhang, S. Sathiya Keerthi, Dhruv Mahajan, Inderjit S. Dhillon, Cho-Jui Hsieh
ICML 2017poster

In this paper, we study the gradient boosted decision trees (GBDT) when the output space is high dimensional and sparse. For example, in multilabel classification, the output space is a $L$-dimensional 0/1 vector, where $L$ is number of labels that can grow to millions and beyond in many modern appl…

Cited by 186Source ↗PDF ↗code ↗Scholar ↗
2016

Asynchronous Parallel Greedy Coordinate Descent

Yang You, Xiangru Lian, Ji Liu, Hsiang-Fu Yu, Inderjit S Dhillon, James Demmel +1
NeurIPS 2016poster

n this paper, we propose and study an Asynchronous parallel Greedy Coordinate Descent (Asy-GCD) algorithm for minimizing a smooth function with bounded constraints. At each iteration, workers asynchronously conduct greedy coordinate descent updates on a block of variables. In the first part of the…

Cited by 54Source ↗PDF ↗Scholar ↗
2016

Dual Decomposed Learning with Factorwise Oracle for Structural SVM of Large Output Domain

Ian En-Hsu Yen, Xiangru Huang, Kai Zhong, Ruohan Zhang, Pradeep K Ravikumar, Inderjit S Dhillon
NeurIPS 2016poster

Many applications of machine learning involve structured output with large domain, where learning of structured predictor is prohibitive due to repetitive calls to expensive inference oracle. In this work, we show that, by decomposing training of Structural Support Vector Machine (SVM) into a series…

Cited by 10Source ↗PDF ↗Scholar ↗
2016

Temporal Regularized Matrix Factorization for High-dimensional Time Series Prediction

Hsiang-Fu Yu, Nikhil Rao, Inderjit S Dhillon
NeurIPS 2016poster

Time series prediction problems are becoming increasingly high-dimensional in modern applications, such as climatology and demand forecasting. For example, in the latter problem, the number of items for which demand needs to be forecast might be as large as 50,000. In addition, the data is generally…

Cited by 638Source ↗PDF ↗Scholar ↗
2015

Collaborative Filtering with Graph Information: Consistency and Scalable Methods

Nikhil Rao, Hsiang-Fu Yu, Pradeep K Ravikumar, Inderjit S Dhillon
NeurIPS 2015spotlight

Low rank matrix completion plays a fundamental role in collaborative filtering applications, the key idea being that the variables lie in a smaller subspace than the ambient space. Often, additional information about the variables is known, and it is reasonable to assume that incorporating this info…

Cited by 302Source ↗PDF ↗code ↗Scholar ↗
2015

Consistent Multilabel Classification

Oluwasanmi O Koyejo, Nagarajan Natarajan, Pradeep K Ravikumar, Inderjit S Dhillon
NeurIPS 2015poster

Multilabel classification is rapidly developing as an important aspect of modern predictive modeling, motivating study of its theoretical aspects. To this end, we propose a framework for constructing and analyzing multilabel classification metrics which reveals novel results on a parametric form for…

Cited by 129Source ↗PDF ↗Scholar ↗
2015

Fixed-Length Poisson MRF: Adding Dependencies to the Multinomial

David I Inouye, Pradeep K Ravikumar, Inderjit S Dhillon
NeurIPS 2015poster

We propose a novel distribution that generalizes the Multinomial distribution to enable dependencies between dimensions. Our novel distribution is based on the parametric form of the Poisson MRF model [Yang et al., 2012] but is fundamentally different because of the domain restriction to a fixed-len…

Cited by 8Source ↗PDF ↗Scholar ↗
2015

Sparse Linear Programming via Primal and Dual Augmented Coordinate Descent

Ian En-Hsu Yen, Kai Zhong, Cho-Jui Hsieh, Pradeep K Ravikumar, Inderjit S Dhillon
NeurIPS 2015poster

Over the past decades, Linear Programming (LP) has been widely used in different areas and considered as one of the mature technologies in numerical optimization. However, the complexity offered by state-of-the-art algorithms (i.e. interior-point method and primal, dual simplex methods) is still uns…

Cited by 40Source ↗PDF ↗Scholar ↗