On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, Siva Reddy
Abstract
Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination. In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehensive human study on both existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Our study reveals that the standard benchmarks consist of > 60% hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research.
BibTeX
@inproceedings{dziri-etal-2022-origin,
title = "On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?",
author = "Dziri, Nouha and
Milton, Sivan and
Yu, Mo and
Zaiane, Osmar and
Reddy, Siva",
editor = "Carpuat, Marine and
de Marneffe, Marie-Catherine and
Meza Ruiz, Ivan Vladimir",
booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jul,
year = "2022",
address = "Seattle, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.naacl-main.387/",
doi = "10.18653/v1/2022.naacl-main.387",
pages = "5271--5285"
}