Can You Share Your Story? Modeling Clients’ Metacognition and Openness for LLM Therapist Evaluation
Minju Kim, Dongje Yoo, Yeonjun Hwang, Minseok Kang, Namyoung Kim, Minju Gwak, Beong-woo Kwak, Hyungjoo Chae
Abstract
Understanding clients’ thoughts and beliefs is fundamental in counseling, yet current evaluations of LLM therapists often fail to assess this ability. Existing evaluation methods rely on client simulators that clearly disclose internal states to the therapist, making it difficult to determine whether an LLM therapist can uncover unexpressed perspectives. To address this limitation, we introduce MindVoyager, a novel evaluation framework featuring a controllable and realistic client simulator which dynamically adapts itself based on the ongoing counseling session, offering a more realistic and challenging evaluation environment. We further introduce evaluation metrics that assess the exploration ability of LLM therapists by measuring their thorough understanding of client’s beliefs and thoughts.
BibTeX
@inproceedings{kim-etal-2025-share-story,
title = "Can You Share Your Story? Modeling Clients' Metacognition and Openness for {LLM} Therapist Evaluation",
author = "Kim, Minju and
Yoo, Dongje and
Hwang, Yeonjun and
Kang, Minseok and
Kim, Namyoung and
Gwak, Minju and
Kwak, Beong-woo and
Chae, Hyungjoo and
Kim, Harim and
Lee, Yunjoong and
Kim, Min Hee and
Jung, Dayi and
Chung, Kyong-Mee and
Yeo, Jinyoung",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.1332/",
doi = "10.18653/v1/2025.findings-acl.1332",
pages = "25943--25962",
ISBN = "979-8-89176-256-5"
}