WebWISE: Unlocking Web Interface Control for LLMs via Sequential Exploration
Heyi Tao, Sethuraman T V, Michal Shlapentokh-Rothman, Tanmay Gupta, Heng Ji, Derek Hoiem
Abstract
This paper investigates using Large Language Models (LLMs) to automatically perform web software tasks using click, scroll, and text in- put operations. Previous approaches, such as reinforcement learning (RL) or imitation learning, are inefficient to train and task-specific. Our method uses filtered Document Object Model (DOM) elements as observations and performs tasks step-by-step, sequentially generating small programs based on the current observations. We use in-context learning, either benefiting from a single manually provided example, or an automatically generated example based on a successful zero-shot trial. We evaluate our proposed method on the MiniWob++ benchmark. With only one in-context example, our WebWISE method using gpt-3.5-turbo achieves similar or better performance than other methods that require many demonstrations or trials.
BibTeX
@inproceedings{tao-etal-2024-webwise,
title = "{W}eb{WISE}: Unlocking Web Interface Control for {LLM}s via Sequential Exploration",
author = "Tao, Heyi and
T V, Sethuraman and
Shlapentokh-Rothman, Michal and
Gupta, Tanmay and
Ji, Heng and
Hoiem, Derek",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-naacl.234/",
doi = "10.18653/v1/2024.findings-naacl.234",
pages = "3693--3711"
}