RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations
Haolan Zhan, Zhuang Li, Xiaoxi Kang, Tao Feng, Yuncheng Hua, Lizhen Qu, Yi Ying, Mei Rianto Chandra
Abstract
Norm violations occur when individuals fail to conform to culturally accepted behaviors, which may lead to potential conflicts. Remediating norm violations requires social awareness and cultural sensitivity of the nuances at play. To equip interactive AI systems with a remediation ability, we offer ReNoVi — a large-scale corpus of 9,258 multi-turn dialogues annotated with social norms, as well as define a sequence of tasks to help understand and remediate norm violations step by step. ReNoVi consists of two parts: 512 human-authored dialogues (real data), and 8,746 synthetic conversations generated by ChatGPT through prompt learning. While collecting sufficient human-authored data is costly, synthetic conversations provide suitable amounts of data to help mitigate the scarcity of training data, as well as the chance to assess the alignment between LLMs and humans in the awareness of social norms. We thus harness the power of ChatGPT to generate synthetic training data for our task. To ensure the quality of both human-authored and synthetic data, we follow a quality control protocol during data collection. Our experimental results demonstrate the importance of remediating norm violations in socio-cultural conversations, as well as the improvement in performance obtained from synthetic data.
BibTeX
@inproceedings{zhan-etal-2024-renovi,
title = "{RENOVI}: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations",
author = "Zhan, Haolan and
Li, Zhuang and
Kang, Xiaoxi and
Feng, Tao and
Hua, Yuncheng and
Qu, Lizhen and
Ying, Yi and
Chandra, Mei Rianto and
Rosalin, Kelly and
Jureynolds, Jureynolds and
Sharma, Suraj and
Qu, Shilin and
Luo, Linhao and
Zukerman, Ingrid and
Soon, Lay-Ki and
Semnani Azad, Zhaleh and
Haf, Reza",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-naacl.196/",
doi = "10.18653/v1/2024.findings-naacl.196",
pages = "3104--3117"
}