Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models
Ying-Chun Lin, Jennifer Neville, Jack Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri
Abstract
Accurate and interpretable user satisfaction estimation (USE) is critical for understanding, evaluating, and continuously improving conversational systems. Users express their satisfaction or dissatisfaction with diverse conversational patterns in both general-purpose (ChatGPT and Bing Copilot) and task-oriented (customer service chatbot) conversational systems. Existing approaches based on featurized ML models or text embeddings fall short in extracting generalizable patterns and are hard to interpret. In this work, we show that LLMs can extract interpretable signals of user satisfaction from their natural language utterances more effectively than embedding-based approaches. Moreover, an LLM can be tailored for USE via an iterative prompting framework using supervision from labeled examples. Our proposed method, Supervised Prompting for User satisfaction Rubrics (SPUR), not only has higher accuracy but is more interpretable as it scores user satisfaction via learned rubrics with a detailed breakdown.
BibTeX
@inproceedings{lin-etal-2024-interpretable,
title = "Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models",
author = "Lin, Ying-Chun and
Neville, Jennifer and
Stokes, Jack and
Yang, Longqi and
Safavi, Tara and
Wan, Mengting and
Counts, Scott and
Suri, Siddharth and
Andersen, Reid and
Xu, Xiaofeng and
Gupta, Deepak and
Jauhar, Sujay Kumar and
Song, Xia and
Buscher, Georg and
Tiwary, Saurabh and
Hecht, Brent and
Teevan, Jaime",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-long.598/",
doi = "10.18653/v1/2024.acl-long.598",
pages = "11100--11115"
}