Love Thy Neighbors: Image Annotation by Exploiting Image Metadata
Justin Johnson, Lamberto Ballan, Li Fei-Fei
Abstract
Some images that are difficult to recognize on their own may become more clear in the context of a neighborhood of related images with similar social-network metadata. We build on this intuition to improve multilabel image annotation. Our model uses image metadata nonparametrically to generate neighborhoods of related images using Jaccard similarities, then uses a deep neural network to blend visual information from the image and its neighbors. Prior work typically models image metadata parametrically; in contrast, our nonparametric treatment allows our model to perform well even when the vocabulary of metadata changes between training and testing. We perform comprehensive experiments on the NUS-WIDE dataset, where we show that our model outperforms state-of-the-art methods for multilabel image annotation even when our model is forced to generalize to new types of metadata.
BibTeX
@inproceedings{iccv2015_lovethyneighbors,
title = {Love Thy Neighbors: Image Annotation by Exploiting Image Metadata},
author = {Justin Johnson and Lamberto Ballan and Li Fei-Fei},
booktitle = {ICCV 2015},
year = {2015}
}