ChatHF: Collecting Rich Human Feedback from Real-time Conversations
Andrew Li, Zhenduo Wang, Ethan Mendes, Duong Minh Le, Wei Xu, Alan Ritter
Abstract
We introduce ChatHF, an interactive annotation framework for chatbot evaluation, which integrates configurable annotation within a chat interface. ChatHF can be flexibly configured to accommodate various chatbot evaluation tasks, for example detecting offensive content, identifying incorrect or misleading information in chatbot responses, and chatbot responses that might compromise privacy. It supports post-editing of chatbot outputs and supports visual inputs, in addition to an optional voice interface. ChatHF is suitable for collection and annotation of NLP datasets, and Human-Computer Interaction studies, as demonstrated in case studies on image geolocation and assisting older adults with daily activities. ChatHF is publicly accessible at https://chat-hf.com.
BibTeX
@inproceedings{li-etal-2024-chathf,
title = "{C}hat{HF}: Collecting Rich Human Feedback from Real-time Conversations",
author = "Li, Andrew and
Wang, Zhenduo and
Mendes, Ethan and
Le, Duong Minh and
Xu, Wei and
Ritter, Alan",
editor = "Hernandez Farias, Delia Irazu and
Hope, Tom and
Li, Manling",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-demo.28/",
doi = "10.18653/v1/2024.emnlp-demo.28",
pages = "270--279"
}