OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use
Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao
Abstract
The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution of multi-modal large language models ((M)LLMs), this dream is closer to reality, as (M)LLM-based Agents using computers, mobile phones and web browsers by operating within the environments and interfaces (e.g., Graphical User Interface (GUI) and Command Line Interface (CLI)) provided by operating systems (OS) to automate tasks have significantly advanced. This paper presents a comprehensive survey on these advanced agents, designated as OS Agents. We begin by elucidating the fundamentals of OS Agents, exploring their key components and capabilities. We then examine methodologies for constructing OS Agents, focusing on domain-specific foundation models and agent frameworks. A detailed review of evaluation metrics and benchmarks highlights how OS Agents are assessed across diverse platforms and tasks. Finally, we discuss current challenges and identify promising directions for future research. An open-source GitHub repository is maintained as a dynamic resource to foster further innovation in this field.
BibTeX
@inproceedings{hu-etal-2025-os,
title = "{OS} Agents: A Survey on {MLLM}-based Agents for Computer, Phone and Browser Use",
author = "Hu, Xueyu and
Xiong, Tao and
Yi, Biao and
Wei, Zishu and
Xiao, Ruixuan and
Chen, Yurun and
Ye, Jiasheng and
Tao, Meiling and
Zhou, Xiangxin and
Zhao, Ziyu and
Li, Yuhuai and
Xu, Shengze and
Wang, Shenzhi and
Xu, Xinchen and
Qiao, Shuofei and
Wang, Zhaokai and
Kuang, Kun and
Zeng, Tieyong and
Wang, Liang and
Li, Jiwei and
Jiang, Yuchen Eleanor and
Zhou, Wangchunshu and
Wang, Guoyin and
Yin, Keting and
Zhao, Zhou and
Yang, Hongxia and
Wu, Fan and
Zhang, Shengyu and
Wu, Fei",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.369/",
doi = "10.18653/v1/2025.acl-long.369",
pages = "7436--7465",
ISBN = "979-8-89176-251-0"
}