Variance Reduction Can Improve Trade-Off in Multi-Objective Learning
Heshan Devaka Fernando, Lisha Chen, Songtao Lu, Pin-Yu Chen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, Gaowen Liu
Abstract
Many machine learning problems today have multiple objective functions, which are often tackled by the multi-objective learning (MOL) framework. Albeit many encouraging results are obtained by MOL algorithms, a recent theoretical study [1] revealed that these gradient-based MOL methods (e.g., MGDA, CAGrad) all reflect an inherent trade-off between optimization convergence speeds and conflict-avoidance abilities. To this end, we develop an improved stochastic variance-reduced multi-objective gradient correction method for MOL, achieving the ${\mathcal{O}}\left({{\varepsilon ^{ - 1.5}}}\right)$ sample complexity. In addition, our proposed method simultaneously improves the theoretical guarantees for conflict avoidance and convergence rate compared to prior stochastic gradient-based MOL methods in the non-convex setting. We further validate the effectiveness of the proposed method empirically using popular multi-task learning (MTL) benchmarks.
BibTeX
@inproceedings{icassp2024_variancereductio,
title = {Variance Reduction Can Improve Trade-Off in Multi-Objective Learning},
author = {Heshan Devaka Fernando and Lisha Chen and Songtao Lu and Pin-Yu Chen and Miao Liu and Subhajit Chaudhury and Keerthiram Murugesan and Gaowen Liu and Meng Wang and Tianyi Chen},
booktitle = {ICASSP 2024},
year = {2024}
}