Out-of-Distribution Detectors: Not Yet Primed for Practical Deployment

Abstract

Out-of-distribution (OoD) detectors work alongside deep neural networks (DNNs) to reduce their risks in eliciting wrong predictions. Unfortunately, OoD detectors built on data-centric designs are also subject to robustness issues, as the DNNs. This paper examines the practical robustness of OoD detectors, taking computer vision tasks as examples and considering natural input perturbations that may come from camera positions and lighting conditions. Our study incorporates extensive experiments over 2000+ settings and correlation studies, highlighting significant challenges in OoD detection robustness, e.g., OoD detectors’ robustness error rate in practical settings can be as high as 28%. The paper advances our understanding of OoD detectors’ applicability in real world and the interplay of their robustness with DNNs’ robustness, calling for novel methodology to design robust OoD detectors in broader signal processing tasks.