Discovering Universal Geometry in Embeddings with ICA
Hiroaki Yamagiwa, Momose Oyama, Hidetoshi Shimodaira
Abstract
This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis (PCA). We demonstrate that each embedding can be expressed as a composition of a few intrinsic interpretable axes and that these semantic axes remain consistent across different languages, algorithms, and modalities. The discovery of a universal semantic structure in the geometric patterns of embeddings enhances our understanding of the representations in embeddings.
BibTeX
@inproceedings{
yamagiwa2023discovering,
title={Discovering Universal Geometry in Embeddings with {ICA}},
author={Hiroaki Yamagiwa and Momose Oyama and Hidetoshi Shimodaira},
booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
year={2023},
url={https://openreview.net/forum?id=iMnwXQemEr}
}