Scientific and Creative Analogies in Pretrained Language Models
Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, Ekaterina Shutova
Abstract
This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.
BibTeX
@inproceedings{czinczoll-etal-2022-scientific,
title = "Scientific and Creative Analogies in Pretrained Language Models",
author = "Czinczoll, Tamara and
Yannakoudakis, Helen and
Mishra, Pushkar and
Shutova, Ekaterina",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-emnlp.153/",
doi = "10.18653/v1/2022.findings-emnlp.153",
pages = "2094--2100"
}