Inconsistency-Based Data-Centric Active Open-Set Annotation
Ruiyu Mao, Ouyang Xu, Yunhui Guo
Abstract
Active learning, a method to reduce labeling effort for training deep neural networks, is often limited by the assumption that all unlabeled data belong to known classes. This closed-world assumption fails in practical scenarios with unknown classes in the data, leading to active open-set annotation challenges. Existing methods struggle with this uncertainty. We introduce NEAT, a novel, computationally efficient, data-centric active learning approach for open-set data. NEAT differentiates and labels known classes from a mix of known and unknown classes, using a clusterability criterion and a consistency mea- sure that detects inconsistencies between model predictions and feature distribution. In contrast to recent learning-centric solutions, NEAT shows superior performance in active open- set annotation, as our experiments confirm. Additional details on the further evaluation metrics, implementation, and archi- tecture of our method can be found in the public document at https://arxiv.org/pdf/2401.04923.pdf.
BibTeX
@article{Mao_Xu_Guo_2024, title={Inconsistency-Based Data-Centric Active Open-Set Annotation}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/28213}, DOI={10.1609/aaai.v38i5.28213}, abstractNote={Active learning, a method to reduce labeling effort for training deep neural networks, is often limited by the assumption that all unlabeled data belong to known classes. This closed-world assumption fails in practical scenarios with unknown classes in the data, leading to active open-set annotation challenges. Existing methods struggle with this uncertainty. We introduce NEAT, a novel, computationally efficient, data-centric active learning approach for open-set data. NEAT differentiates and labels known classes from a mix of known and unknown classes, using a clusterability criterion and a consistency mea- sure that detects inconsistencies between model predictions and feature distribution. In contrast to recent learning-centric solutions, NEAT shows superior performance in active open- set annotation, as our experiments confirm. Additional details on the further evaluation metrics, implementation, and archi- tecture of our method can be found in the public document at https://arxiv.org/pdf/2401.04923.pdf.}, number={5}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Mao, Ruiyu and Xu, Ouyang and Guo, Yunhui}, year={2024}, month={Mar.}, pages={4180-4188} }