PairScale: Analyzing Attitude Change with Pairwise Comparisons
Rupak Sarkar, Patrick Y. Wu, Kristina Miler, Alexander Miserlis Hoyle, Philip Resnik
Abstract
We introduce a text-based framework for measuring attitudes in communities toward issues of interest, going beyond the pro/con/neutral of conventional stance detection to characterize attitudes on a continuous scale using both implicit and explicit evidence in language. The framework exploits LLMs both to extract attitude-related evidence and to perform pairwise comparisons that yield unidimensional attitude scores via the classic Bradley-Terry model. We validate the LLM-based steps using human judgments, and illustrate the utility of the approach for social science by examining the evolution of attitudes on two high-profile issues in U.S. politics in two political communities on Reddit over the period spanning from the 2016 presidential campaign to the 2022 mid-term elections. WARNING: Potentially sensitive political content.
BibTeX
@inproceedings{sarkar-etal-2025-pairscale,
title = "{P}air{S}cale: Analyzing Attitude Change with Pairwise Comparisons",
author = "Sarkar, Rupak and
Wu, Patrick Y. and
Miler, Kristina and
Hoyle, Alexander Miserlis and
Resnik, Philip",
editor = "Chiruzzo, Luis and
Ritter, Alan and
Wang, Lu",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-naacl.94/",
pages = "1722--1738",
ISBN = "979-8-89176-195-7"
}