Segmentation and Shape Estimation of Multiple Deformed Cloths Using a CNN-Based Landmark Detector and Clustering

Abstract

In this paper, we propose a method for segmentation and shape estimation of multiple deformed cloths stacked on a floor from an image using a CNN-based landmark detector and clustering. The proposed method first estimates landmark positions from the heatmaps generated from the landmark detector and then clusters them using their attributes also given from the landmark detector. The contributions of the proposed method are twofold: (1) it can perform segmentation and shape estimation of multiple cloths of the same type or different types simultaneously by modeling a cloth as a grid of landmarks and estimating their positions and attributes, (2) it can correctly segment severely occluded or seemingly disjointed cloths by estimating and classifying landmark attributes including grid index, cloth position, cloth orientation, and cloth type. We evaluate the proposed method on both synthetic and real image datasets and show that the proposed method outperforms three baseline methods. We also show that the proposed method enables our humanoid robot to pick up the desired type of cloth from a pile of cloths.