AlignBench: Benchmarking Chinese Alignment of Large Language Models
Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke
Abstract
Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese. We tailor a human-in-the-loop data curation pipeline, containing 8 main categories, 683 real-scenario rooted queries and corresponding human verified references.To ensure references’ correctness, each knowledge-intensive query is accompanied with evidences collected from reliable webpages (including the url and quotation) by our annotators.For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge (CITATION) with Chain-of-Thought to generate explanations and final ratings as evaluations, ensuring high reliability and interpretability.All evaluation codes and data are publicly available at https://github.com/THUDM/AlignBench
BibTeX
@inproceedings{liu-etal-2024-alignbench,
title = "{A}lign{B}ench: Benchmarking {C}hinese Alignment of Large Language Models",
author = "Liu, Xiao and
Lei, Xuanyu and
Wang, Shengyuan and
Huang, Yue and
Feng, Andrew and
Wen, Bosi and
Cheng, Jiale and
Ke, Pei and
Xu, Yifan and
Tam, Weng Lam and
Zhang, Xiaohan and
Sun, Lichao and
Gu, Xiaotao and
Wang, Hongning and
Zhang, Jing and
Huang, Minlie and
Dong, Yuxiao and
Tang, Jie",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-long.624/",
doi = "10.18653/v1/2024.acl-long.624",
pages = "11621--11640"
}