Yongwei Che

The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities

While internet-scale data often come in pairs (e.g., audio+image, image+text), we often want to perform inferences over modalities unseen together in the training data (e.g., audio+text). Prior work has addressed this issue by learning multiple contrastive embedding spaces between existing modality…