GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models
Dylan Hutson, Daniel Vennemeyer, Aneesh Deshmukh, Justin Zhan, Tianyu Jiang
Abstract
We introduce GuessingGame, a protocol for evaluating large language models (LLMs) as strategic question-askers in open-ended, open-domain settings. A Guesser LLM identifies a hidden object by posing free-form questions to an Oracle—without predefined choices or candidate lists. To measure question quality, we propose two information gain (IG) metrics: a Bayesian method that tracks belief updates over semantic concepts using LLM-scored relevance, and an entropy-based method that filters candidates via ConceptNet. Both metrics are model-agnostic and support post hoc analysis. Across 858 games with multiple models and prompting strategies, higher IG strongly predicts efficiency: a one-standard-deviation IG increase reduces expected game length by 43%. Prompting constraints guided by IG—such as enforcing question diversity—enable weaker models to match GPT-4o. These results show that question-asking in LLMs is both measurable and improvable, and crucial for interactive reasoning.
BibTeX
@inproceedings{emnlp2025_guessinggamemeas,
title = {GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models},
author = {Dylan Hutson and Daniel Vennemeyer and Aneesh Deshmukh and Justin Zhan and Tianyu Jiang},
booktitle = {EMNLP 2025},
year = {2025}
}