Embodied Understanding of Driving Scenarios
Yunsong Zhou*, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo
Abstract
"Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Such understanding is typically founded upon Vision-Language Models (VLMs). Nevertheless, existing VLMs are restricted to the 2D domain, devoid of spatial awareness and long-horizon extrapolation proficiencies. We revisit the key aspects of autonomous driving and formulate appropriate rubrics. Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents’ understanding of driving scenes with large spatial and temporal spans. ELM incorporates space-aware pre-training to endow the agent with robust spatial localization capabilities. Besides, the model employs time-aware token selection to accurately inquire about temporal cues. We instantiate ELM on the reformulated multi-faced benchmark, and it surpasses previous state-of-the-art approaches in all aspects. All code, data, and models are accessible at https://github.com/OpenDriveLab/ELM."
BibTeX
@inproceedings{eccv2024_embodiedundersta,
title = {Embodied Understanding of Driving Scenarios},
author = {Yunsong Zhou* and Linyan Huang and Qingwen Bu and Jia Zeng and Tianyu Li and Hang Qiu and Hongzi Zhu and Minyi Guo and Yu Qiao and Hongyang Li},
booktitle = {ECCV 2024},
year = {2024}
}