When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels
Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, Jing Xu
Abstract
Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.
BibTeX
@inproceedings{shi-etal-2024-life,
title = "When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels",
author = "Shi, Weiyan and
Dinan, Emily and
Shuster, Kurt and
Weston, Jason and
Xu, Jing",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.naacl-long.169/",
doi = "10.18653/v1/2024.naacl-long.169",
pages = "3066--3082"
}