$\texttt{MetaDistill}$: Unlocking the Performance Ceiling for Pretrained Optimizers

Abstract

Meta Black-Box Optimization (MetaBBO) has emerged as a promising paradigm by employing meta learning to automatically optimize the configurations of low-level black-box optimizers. Despite its potential, the generalization of MetaBBO remains significantly constrained when facing unseen, complex objective landscapes. We identify that this bottleneck stems from a restricted performance upper bound inherent in current training mechanisms: by learning from scratch in a self-supervised or unsupervised manner, meta optimizers are never exposed to advanced, high-quality optimization behaviors, forcing them to converge on suboptimal strategies. In this paper, we propose $\texttt{MetaDistill}$, a general MetaBBO training framework designed to lift the strategy ceiling through pretraining and test-time fine-tuning. In the pretraining stage, we represent high-quality strategies from classical algorithms as expert optimization trajectories and utilize them for diversity-preserving distillation, enabling the learnable optimizer to internalize advanced optimization behaviors. In the optional fine-tuning stage, we perform self-supervised fine-tuning as a warm-start procedure to further refine the distilled knowledge on unseen tasks. We evaluate our $\texttt{MetaDistill}$ framework on the BBOB test suite and three control tasks. The results demonstrate that $\texttt{MetaDistill}$ significantly improves the generalization ability of various learnable optimizers compared to their original training paradigms.