Generating Vehicular Icon Descriptions and Indications Using Large Vision-Language Models
James Fletcher, Nicholas Dehnen, Seyed Nima Tayarani Bathaie, Aijun An, Heidar Davoudi, Ron DiCarlantonio, Gary Farmaner
Abstract
To enhance a question-answering system for automotive drivers, we tackle the problem of automatic generation of icon image descriptions. The descriptions can match the driver’s query about the icon appearing on the dashboard and tell the driver what is happening so that they may take an appropriate action. We use three state-of-the-art large vision-language models to generate both visual and functional descriptions based on the icon image and its context information in the car manual. Both zero-shot and few-shot prompts are used. We create a dataset containing over 400 icons with their ground-truth descriptions and use it to evaluate model-generated descriptions across several performance metrics. Our evaluation shows that two of these models (GPT-4o and Claude 3.5) performed well on this task, while the third model (LLaVA-NEXT) performs poorly.
BibTeX
@inproceedings{fletcher-etal-2024-generating,
title = "Generating Vehicular Icon Descriptions and Indications Using Large Vision-Language Models",
author = "Fletcher, James and
Dehnen, Nicholas and
Tayarani Bathaie, Seyed Nima and
An, Aijun and
Davoudi, Heidar and
DiCarlantonio, Ron and
Farmaner, Gary",
editor = "Dernoncourt, Franck and
Preo{\c{t}}iuc-Pietro, Daniel and
Shimorina, Anastasia",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
month = nov,
year = "2024",
address = "Miami, Florida, US",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-industry.83/",
doi = "10.18653/v1/2024.emnlp-industry.83",
pages = "1107--1120"
}