Evaluating Large Language Models for Belief Inference: Mapping Belief Networks at Scale
Trisevgeni Papakonstantinou, Antonina Zhiteneva, Ana Yutong Ma, Derek Powell, Zachary Horne
Abstract
Beliefs are interconnected, influencing how people process and update what they think. To study the interconnectedness of beliefs at scale, we introduce a novel analytical pipeline leveraging a finetuned GPT-4o model to infer belief structures from large-scale social media data. We evaluate the model’s performance by (1) comparing it to human annotated data (2) and its inferences to human-generated survey data. Our results show that a fine-tuned GPT-4o model can effectively recover belief structures, allowing for a level of scalability and efficiency that is impossible using traditional survey methods of data collection. This work demonstrates the potential for large language models to perform belief inference tasks and provides a framework for future research on the analysis of belief structures.
BibTeX
@inproceedings{emnlp2025_evaluatinglargel,
title = {Evaluating Large Language Models for Belief Inference: Mapping Belief Networks at Scale},
author = {Trisevgeni Papakonstantinou and Antonina Zhiteneva and Ana Yutong Ma and Derek Powell and Zachary Horne},
booktitle = {EMNLP 2025},
year = {2025}
}