Felicity Wang

Numerical Optimizations for Weighted Low-rank Estimation on Language Models

Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. The parameters of a trained neu…