Black-Box Assisted Regression: Phase Transitions and Minimax Optimality

Abstract

Foundation models provide strong but biased priors for downstream tasks with limited labeled data. We formalize black-box assisted nonparametric regression where $\|f^*-f_0\|_{L_2(P_X)}\le\delta$ for unknown $\delta$. We characterize the minimax risk, revealing a phase transition at $\delta_c(n)\asymp n^{-\beta/(2\beta+d)}$ with optimal rate $\min\{\delta^2, n^{-2\beta/(2\beta+d)}\}$. Our Safe Black-Box Residual Estimator integrates zero-initialization (a strong inductive bias) with a holdout selection mechanism. This approach achieves the minimax rate while avoiding negative transfer. Experiments on synthetic data, CIFAR-100 (CLIP), and AG News (Qwen3-8B) validate the theory, improving sample efficiency (e.g., CIFAR-100 at $n=2000$: from 59.4\% zero-shot to 66.7\%).