Understanding and Enhancing Differentiable Architecture Search from Information Bottleneck Perspective
Haidong Kang, Lianbo Ma, Pengjun Chen, Qiang He, Bo Yi
Abstract
Performance collapse is an intractable issue of Differentiable Architecture Search (DAS), where severe performance degradation of DAS happens when it trains on different search spaces or datasets. We theoretically analyze the issue from the information bottleneck (IB) perspective, and disclose that a solution to overcome this problem is to seek the bifurcation point of IB tradeoff between compression and prediction of the supernet. To this end, we propose a simple yet highly effective method, namely, Batch Entropy-decay Regularization (BER), to guide the learning of DAS, which restricts compression in DAS by imposing a penalty on the architecture parameters. Comprehensive theoretical analyses demonstrate that BER is able to completely resolve DAS
BibTeX
@inproceedings{aaai2026_understandingand,
title = {Understanding and Enhancing Differentiable Architecture Search from Information Bottleneck Perspective},
author = {Haidong Kang and Lianbo Ma and Pengjun Chen and Qiang He and Bo Yi},
booktitle = {AAAI 2026},
year = {2026}
}