Jointly Predicting Future Sequence and Steering Angles for Dynamic Driving Scenes

Abstract

Generative Adversarial Network (GAN) has attracted rising attention for video future sequence prediction in driving scenes. However, the images generated by GAN often miss the target for lack of any constraints for its generated target. In this paper, an encoder-decoder based multi-task video prediction network - SegVAE is proposed by simultaneously accomplishing the predictions (generations) of both future sequence and steering angles for egocentric driving videos at pixel-level. Specifically, the encoder is constructed based on Varitional Auto-Encoder (VAE) to learn the complex latent distribution of real driving scenes. The decoder is exploited with a multi-task manner to jointly predict the future sequence and steering angles of dynamic driving scenes, where an enhanced generation mechanism is also proposed. Varitional Auto-Encoder (VAE) and Long Short Term Memory Networks (LSTM) are introduced to optimize the learning of SegVAE. The experimental results on public KITTI and NVIDIA driving datasets indicate that the proposed Seg-VAE can effectively mimic humans prediction mechanism, and outperform standard VAE and CNN-based generative adversarial network.