On the Fine-Grained Planning Abilities of VLM Web Agents
Surgan Jandial, Yinong Oliver Wang, Andrea Bajcsy, Fernando De la Torre
Abstract
Vision-Language Models (VLMs) have shown promise as web agents, yet their planning—the ability to devise strategies or action sequences to complete tasks—remains understudied. While prior works focus on VLM’s perception and overall success rates (i.e., goal completion), fine-grained investigation of their planning has been overlooked. To address this gap, we examine VLMs’ capability to (1) understand temporal relationships within web contexts, and (2) assess plans of actions across diverse scenarios. We design four simple yet effective tests to delve into these nuanced aspects around planning. Our results across nineteen VLMs reveal that these models exhibit limited performance in the aforementioned skills and are not reliable to function as web agents. To facilitate future work, we release our planning evaluations and data, providing a foundation for advancing the future research in this area.
BibTeX
@inproceedings{emnlp2025_onthefinegrained,
title = {On the Fine-Grained Planning Abilities of VLM Web Agents},
author = {Surgan Jandial and Yinong Oliver Wang and Andrea Bajcsy and Fernando De la Torre},
booktitle = {EMNLP 2025},
year = {2025}
}