Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, Antonio Torralba
Abstract
In this paper, we introduce Recipe1M, a new large-scale, structured corpus of over 1m cooking recipes and 800k food images. As the largest publicly available collection of recipe data, Recipe1M affords the ability to train high-capacity models on aligned, multi-modal data. Accordingly, we train a neural network to find a joint embedding of recipes and images that yields impressive results on an image-recipe retrieval task. Additionally, we demonstrate that regularization via the addition of a high-level, semantic classification objective improves performance to rival that of humans and enables semantic vector arithmetic. We postulate that these embeddings will provide a basis for further exploration of the Recipe1M dataset and food and cooking in general.
BibTeX
@inproceedings{cvpr2017_learningcrossmod,
title = {Learning Cross-Modal Embeddings for Cooking Recipes and Food Images},
author = {Amaia Salvador and Nicholas Hynes and Yusuf Aytar and Javier Marin and Ferda Ofli and Ingmar Weber and Antonio Torralba},
booktitle = {CVPR 2017},
year = {2017}
}