HID-NAS: A Novel Neural Architecture Search Pipeline for High Information Density Data

Abstract

Neural Architecture Search (NAS) is a core component of automated machine learning, enabling the automatic discovery of task-specific architectures. However, directly employing raw data for NAS faces challenges in terms of large data volume, low search efficiency, and potential data privacy concerns. To address these limitations, we condense large-scale target datasets into high information-density proxy datasets. And we propose a novel pipeline, HID-NAS, which skillfully exploits High Information Density(HID) data for efficient neural architecture search. At the algorithmic level, our algorithm focuses on the inherent problem of DARTS-derived algorithms, i.e., the discrepancy between trained supernet and selected architectures. We pioneer the phenomenon of "Selection Error" that occurs during architecture selection and introduce an attention mechanism that focuses on candidate edges to improve the accuracy of pipeline-selected architectures. Furthermore, we also formulate a regularization mechanism that employs the high information density data of the proxy dataset to transform the architectural parameter updates into an adversarial game and adjust the strengths through a normalization mechanism. These three key components – attention, regularization, and normalization – allow our pipeline to efficiently identify high-quality models using the distilled dataset. Experimental results demonstrate a significant reduction in search time, achieving high-quality models within just two minutes.