KABON: Knowledge Aggregation with Vision-Language Model for Black-Box Open-Set Domain Adaptation

Abstract

In this paper, we aim to tackle the challenging Black-Box Open-Set Domain Adaptation (BB-OSDA) task. BB-OSDA enables conducting Open-Set Domain Adaptation (OSDA) with solely a black-box source model, broadening the application scope of OSDA. Inspired by the significant success of pre-trained large vision-language (ViL) models in various applications, we propose a novel method, termed Knowledge Aggregation for Black-box Open-set domain adaptatioN (KABON), which leverages the power of ViL models to solve the BB-OSDA problem. Specifically, we first devise a novel knowledge aggregation approach to harness both the generic knowledge from the ViL model and the task-specific knowledge from the black-box source model. Subsequently, we utilize a Gaussian Mixture Model (GMM) with entropy criterion to divide samples of target domain into shared or novel classes. Furthermore, a self-correction strategy is proposed to refine the division of shared and novel classes. Finally, we leverage the divided samples through entropy minmax learning to simultaneously achieve shared classes adaptation and novel classes detection. Experiments conducted on multiple benchmark datasets demonstrate the effectiveness of our proposed method.