Robust Detection for Cluster Analysis

Abstract

The problem of deciding whether a given set of data points forms one cluster or two clusters is investigated from a robust hypothesis testing perspective. It is assumed that a clustering algorithm exists that for both cases calculates cluster assignments and estimates of the corresponding probability density functions. Based on the latter, a statistical hypothesis test for the true number of clusters is formulated. In order to take falsely labeled data points into account, the clusters are then modeled as being contaminated with outliers. This leads to an uncertainty model for the cluster densities of the ε-contamination type, whose corresponding minimax optimal robust detector is well-known and can be implemented using least favorable densities. The performance of this detector under cluster overlap, cluster imbalance, and for different contamination ratios is evaluated numerically and is compared to that of a Bayesian cluster enumeration criterion. Significant performance improvements are shown in all cases.