Amanda Swearngin — accepted AI-conference papers

2025

UINavBench: A Framework for Comprehensive Evaluation of Interactive Digital Agents

Harsh Agrawal, Eldon Schoop, Xinlei Pan, Anuj Mahajan, Ari Seff, Di Feng +9

ICCV 2025poster

We build a comprehensive online evaluation benchmark for language-conditioned multi-step task execution on mobile interfaces. Our benchmark strives to evaluate the multi-step planning, reasoning, and visual grounding capabilities of agents, using mobile user interfaces as a concrete testbed. To buil…

Cited by 0 Source ↗PDF ↗Scholar ↗

2024

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

Keen You*, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeff Nichols +2

ECCV 2024poster

"Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced unde…

Cited by 109 Source ↗PDF ↗Scholar ↗