Visual-Linguistic Reasoning for Pedestrian Trajectory Prediction

Abstract

Accurate prediction of pedestrian trajectories is crucial as autonomous vehicles become more prevalent on roads. The dynamic nature of urban environments and the less predictable behavior of pedestrians present significant challenges in developing reliable prediction models. Earlier methods relying on recurrent neural networks (RNNs) and long-shortterm memory (LSTM) networks have shown promise, but often fail to fully take advantage of the rich visual and contextual information available in real-world scenarios. Recent advances in vision-language models (VLMs) offer new opportunities to improve pedestrian trajectory prediction by incorporating multimodal reasoning capabilities. This paper introduces a novel approach that uses a powerful pre-trained VLM to improve the estimation of pedestrian trajectories. Specifically, we first enable learning of semantically useful scene context and high-level reasoning features via vision-language model finetuning on specific prompts using road scenes with pedestrians. Next, with the learned VLM features and the pedestrian's past trajectory history, we predict future trajectories using an encoder-decoder head. Through experiments with first-person datasets JAAD and PIE, we show that utilizing visual-linguistic semantics via a pretrained vision-language model outperforms previous methods in both deterministic and stochastic trajectory prediction setups.