Position: Rethinking LLM Bias Probing Using Lessons from the Social Sciences
Kirsten Morehouse, Siddharth Swaroop, Weiwei Pan
Abstract
The proliferation of LLM bias probes introduces three challenges: we lack (1) principled criteria for selecting appropriate probes, (2) a system for reconciling conflicting results across probes, and (3) formal frameworks for reasoning about when and why experimental findings will generalize to real user behavior. In response, we propose a systematic approach to LLM social bias probing, drawing on insights from the social sciences. Central to this approach is EcoLevels—a novel framework that helps (a) identify appropriate bias probes (b) reconcile conflicting results, and (c) generate predictions about bias generalization. We ground our framework in the social sciences, as many LLM probes are adapted from human studies, and these fields have faced similar challenges when studying bias in humans. Finally, we outline five lessons that demonstrate how LLM bias probing can (and should) benefit from decades of social science research
BibTeX
@inproceedings{
morehouse2025position,
title={Position: Rethinking {LLM} Bias Probing Using Lessons from the Social Sciences},
author={Kirsten Morehouse and Siddharth Swaroop and Weiwei Pan},
booktitle={Forty-second International Conference on Machine Learning Position Paper Track},
year={2025},
url={https://openreview.net/forum?id=tctWi7I5wd}
}