Temporal information integration for video semantic segmentation

Abstract

We present a temporal Bayesian filter for semantic segmentation of a video sequence. Each pixel is a random variable following a discrete probabilistic distribution function representing possible semantic classes. Bayesian filtering consists in two main steps: 1) a prediction model and 2) an observation model (likelihood). We propose to use a datadriven prediction function derived from a dense optical flow between images t and t + 1 achieved by a deep neural network [1]. Moreover, the observation function uses a semantic segmentation network. The resulting approach is evaluated on the public dataset Cityscapes. We show that using the temporal filtering increases the accuracy of the semantic segmentation.