Hyunsuk Ko

Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-tuning

We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite differences using random perturbations, often suffer from high variance and suboptimal search directions. Our approach ad