The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho
Abstract
As language models (LMs) become capable of handling a wide range of tasks, their evaluation is becoming as challenging as their development. Most generation benchmarks currently assess LMs using abstract evaluation criteria-like helpfulness and harmlessness-which often lack the flexibility and granularity of human assessment. Additionally, these benchmarks tend to focus disproportionately on specific capabilities such as instruction following, leading to coverage bias. To overcome these limitations, we introduce the BiGGen Bench, a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation. We apply this benchmark to assess 100 frontier LMs using five evaluator LMs. Our code, data, and evaluation results are all publicly available at https://github.com/prometheus-eval/prometheus-eval.
BibTeX
@inproceedings{kim-etal-2025-biggen,
title = "The {B}i{GG}en Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models",
author = "Kim, Seungone and
Suk, Juyoung and
Cho, Ji Yong and
Longpre, Shayne and
Kim, Chaeeun and
Yoon, Dongkeun and
Son, Guijin and
Cho, Yejin and
Shafayat, Sheikh and
Baek, Jinheon and
Park, Sue Hyun and
Hwang, Hyeonbin and
Jo, Jinkyung and
Cho, Hyowon and
Shin, Haebin and
Lee, Seongyun and
Oh, Hanseok and
Lee, Noah and
Ho, Namgyu and
Joo, Se June and
Ko, Miyoung and
Lee, Yoonjoo and
Chae, Hyungjoo and
Shin, Jamin and
Jang, Joel and
Ye, Seonghyeon and
Lin, Bill Yuchen and
Welleck, Sean and
Neubig, Graham and
Lee, Moontae and
Lee, Kyungjae and
Seo, Minjoon",
editor = "Chiruzzo, Luis and
Ritter, Alan and
Wang, Lu",
booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.naacl-long.303/",
pages = "5877--5919",
ISBN = "979-8-89176-189-6"
}