A Dataset for Expert Reviewer Recommendation with Large Language Models as Zero-shot Rankers
Vanja M. Karan, Stephen McQuistin, Ryo Yanagida, Colin Perkins, Gareth Tyson, Ignacio Castro, Patrick G.T. Healey, Matthew Purver
Abstract
The task of reviewer recommendation is increasingly important, with main techniques utilizing general models of text relevance. However, state of the art (SotA) systems still have relatively high error rates. Two possible reasons for this are: a lack of large datasets and the fact that large language models (LLMs) have not yet been applied. To fill these gaps, we first create a substantial new dataset, in the domain of Internet specification documents; then we introduce the use of LLMs and evaluate their performance. We find that LLMs with prompting can improve on SotA in some cases, but that they are not a cure-all: this task provides a challenging setting for prompt-based methods
BibTeX
@inproceedings{karan-etal-2025-dataset,
title = "A Dataset for Expert Reviewer Recommendation with Large Language Models as Zero-shot Rankers",
author = "Karan, Vanja M. and
McQuistin, Stephen and
Yanagida, Ryo and
Perkins, Colin and
Tyson, Gareth and
Castro, Ignacio and
Healey, Patrick G.T. and
Purver, Matthew",
editor = "Rambow, Owen and
Wanner, Leo and
Apidianaki, Marianna and
Al-Khalifa, Hend and
Eugenio, Barbara Di and
Schockaert, Steven",
booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
month = jan,
year = "2025",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.coling-main.756/",
pages = "11422--11427"
}