Hyperbolic Image-text Representations
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, Shanmukha Ramakrishna Vedantam
Abstract
Visual and linguistic concepts naturally organize themselves in a hierarchy, where a textual concept "dog" entails all images that contain dogs. Despite being intuitive, current large-scale vision and language models such as CLIP do not explicitly capture such hierarchy. We propose MERU, a contrastive model that yields hyperbolic representations of images and text. Hyperbolic spaces have suitable geometric properties to embed tree-like data, so MERU can better capture the underlying hierarchy in image-text datasets. Our results show that MERU learns a highly interpretable and structured representation space while being competitive with CLIP's performance on standard multi-modal tasks like image classification and image-text retrieval.
BibTeX
@inproceedings{icml2023_hyperbolicimaget,
title = {Hyperbolic Image-text Representations},
author = {Karan Desai and Maximilian Nickel and Tanmay Rajpurohit and Justin Johnson and Shanmukha Ramakrishna Vedantam},
booktitle = {ICML 2023},
year = {2023}
}