ACL 2025finding0 citations
Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations
Jiho Jin, Woosung Kang, Junho Myung, Alice Oh
Abstract
Measuring social bias in large language models (LLMs) is crucial, but existing bias evaluation methods struggle to assess bias in long-form generation. We propose a Bias Benchmark for Generation (BBG), an adaptation of the Bias Benchmark for QA (BBQ), designed to evaluate social bias in long-form generation by having LLMs generate continuations of story prompts. Building our benchmark in English and Korean, we measure the probability of neutral and biased generations across ten LLMs. We also compare our long-form story generation evaluation results with multiple-choice BBQ evaluation, showing that the two approaches produce inconsistent results.
BibTeX
@inproceedings{jin-etal-2025-social,
title = "Social Bias Benchmark for Generation: A Comparison of Generation and {QA}-Based Evaluations",
author = "Jin, Jiho and
Kang, Woosung and
Myung, Junho and
Oh, Alice",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.585/",
doi = "10.18653/v1/2025.findings-acl.585",
pages = "11215--11228",
ISBN = "979-8-89176-256-5"
}