2026
AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning
ICML 2026poster
Zeroth-Order (ZO) optimization has emerged as a promising solution for fine-tuning LLMs under strict memory constraints, as it avoids the prohibitive memory cost of storing activations for backpropagation. However, existing ZO methods typically employ isotropic perturbations, neglecting the rich str…