Ethan Nguyen

Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training

Parameter-efficient training, based on low-rank optimization, has become a highly successful tool for fine-tuning large deep-learning models. However, these methods fail at low-rank pre-training tasks where maintaining the low-rank structure and the objective remains a challenging task. We propose t…