GUI Agents: A Survey
Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu
Abstract
Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed.
BibTeX
@inproceedings{nguyen-etal-2025-gui,
title = "{GUI} Agents: A Survey",
author = "Nguyen, Dang and
Chen, Jian and
Wang, Yu and
Wu, Gang and
Park, Namyong and
Hu, Zhengmian and
Lyu, Hanjia and
Wu, Junda and
Aponte, Ryan and
Xia, Yu and
Li, Xintong and
Shi, Jing and
Chen, Hongjie and
Lai, Viet Dac and
Xie, Zhouhang and
Kim, Sungchul and
Zhang, Ruiyi and
Yu, Tong and
Tanjim, Mehrab and
Ahmed, Nesreen K. and
Mathur, Puneet and
Yoon, Seunghyun and
Yao, Lina and
Kveton, Branislav and
Kil, Jihyung and
Nguyen, Thien Huu and
Bui, Trung and
Zhou, Tianyi and
Rossi, Ryan A. and
Dernoncourt, Franck",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.1158/",
doi = "10.18653/v1/2025.findings-acl.1158",
pages = "22522--22538",
ISBN = "979-8-89176-256-5"
}