An Interactive Evaluation Framework for Empathetic Response Generation

Abstract

Empathetic response generation is a significant domain in Natural Language Processing (NLP). Its development is a critical step toward achieving humanized AI systems. However, current evaluations of empathetic dialogue models are primarily single-turn and static, leading to bias between evaluation results and real-world multi-turn interaction performance. To overcome the longstanding challenge, we propose a novel Interactive Empathy Evaluation Framework (IEEF). It eliminates the bias by facilitating a human-free multi-turn interaction evaluation. Specifically, for human-free interaction, we design a user simulator using reinforcement learning, leveraging a reward model based on LLM scoring. For evalution, we introduce a series of empathy-related metrics based on LLM. The experiments show that IEEF’s evaluation results are highly correlated with real-world multi-turn interaction performance, demonstrating its alignment with human preferences in empathy evaluation.