IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces
Kelly Marchisio, Neha Verma, Kevin Duh, Philipp Koehn
Abstract
The ability to extract high-quality translation dictionaries from monolingual word embedding spaces depends critically on the geometric similarity of the spaces—their degree of “isomorphism.” We address the root-cause of faulty cross-lingual mapping: that word embedding training resulted in the underlying spaces being non-isomorphic. We incorporate global measures of isomorphism directly into the skipgram loss function, successfully increasing the relative isomorphism of trained word embedding spaces and improving their ability to be mapped to a shared cross-lingual space. The result is improved bilingual lexicon induction in general data conditions, under domain mismatch, and with training algorithm dissimilarities. We release IsoVec at https://github.com/kellymarchisio/isovec.
BibTeX
@inproceedings{marchisio-etal-2022-isovec,
title = "{I}so{V}ec: Controlling the Relative Isomorphism of Word Embedding Spaces",
author = "Marchisio, Kelly and
Verma, Neha and
Duh, Kevin and
Koehn, Philipp",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.404/",
doi = "10.18653/v1/2022.emnlp-main.404",
pages = "6019--6033"
}