Chengxun Jin

LOZO+: Provably Efficient Zeroth-Order Fine-Tuning via Greedy Low-Rank Subspace Selection

Zeroth-order (ZO) optimization offers a more memory-efficient alternative to first-order methods for fine-tuning large language models (LLMs). Recent ZO methods, exemplified by LOZO, estimate gradients within low-rank subspaces to align with the low-rank structure of LLM gradients. However, these me…