Can Transformers Reason in Fragments of Natural Language?
Viktor Schlegel, Kamen Pavlov, Ian Pratt-Hartmann
Abstract
State-of-the-art deep-learning-based approaches to Natural Language Processing (NLP) are credited with various capabilities that involve reasoning with natural language texts. %However, reasoning in this setting is often ill-defined and shallow. In this paper we carry out a large-scale empirical study investigating the detection of formally valid inferences in controlled fragments of natural language for which the satisfiability problem becomes increasingly complex. We find that, while transformer-based language models perform surprisingly well in these scenarios, a deeper analysis reveals that they appear to overfit to superficial patterns in the data rather than acquiring the logical principles governing the reasoning in these fragments.
BibTeX
@inproceedings{schlegel-etal-2022-transformers,
title = "Can Transformers Reason in Fragments of Natural Language?",
author = "Schlegel, Viktor and
Pavlov, Kamen and
Pratt-Hartmann, Ian",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.768/",
doi = "10.18653/v1/2022.emnlp-main.768",
pages = "11184--11199"
}