SPHERE: An Evaluation Card for Human-AI Systems
Dora Zhao, Qianou Ma, Xinran Zhao, Chenglei Si, Chenyang Yang, Ryan Louie, Ehud Reiter, Diyi Yang
Abstract
In the era of Large Language Models (LLMs), establishing effective evaluation methods and standards for diverse human-AI interaction systems is increasingly challenging. To encourage more transparent documentation and facilitate discussion on human-AI system evaluation design options, we present an evaluation card SPHERE, which encompasses five key dimensions: 1) What is being evaluated?; 2) How is the evaluation conducted?; 3) Who is participating in the evaluation?; 4) When is evaluation conducted?; 5) How is evaluation validated? We conduct a review of 39 human-AI systems using SPHERE, outlining current evaluation practices and areas for improvement. We provide three recommendations for improving the validity and rigor of evaluation practices.
BibTeX
@inproceedings{zhao-etal-2025-sphere,
title = "{SPHERE}: An Evaluation Card for Human-{AI} Systems",
author = "Zhao, Dora and
Ma, Qianou and
Zhao, Xinran and
Si, Chenglei and
Yang, Chenyang and
Louie, Ryan and
Reiter, Ehud and
Yang, Diyi and
Wu, Tongshuang",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.70/",
doi = "10.18653/v1/2025.findings-acl.70",
pages = "1340--1365",
ISBN = "979-8-89176-256-5"
}