NeurIPS 2021poster19 citations
COHESIV: Contrastive Object and Hand Embedding Segmentation In Video
Dandan Shan, Richard Ely Locke Higgins, David Fouhey
Abstract
In this paper we learn to segment hands and hand-held objects from motion. Our system takes a single RGB image and hand location as input to segment the hand and hand-held object. For learning, we generate responsibility maps that show how well a hand's motion explains other pixels' motion in video. We use these responsibility maps as pseudo-labels to train a weakly-supervised neural network using an attention-based similarity loss and contrastive loss. Our system outperforms alternate methods, achieving good performance on the 100DOH, EPIC-KITCHENS, and HO3D datasets.
Hand Object InteractionObject SegmentationContrastive LearningAttentionEmbeddings
BibTeX
@inproceedings{
shan2021cohesiv,
title={{COHESIV}: Contrastive Object and Hand Embedding Segmentation In Video},
author={Dandan Shan and Richard Ely Locke Higgins and David Fouhey},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=D-ti-5lgbG}
}