Pixel-level Texture Segmentation Based AV1 Video Compression

Abstract

Modern video coding standards use hybrid coding techniques to remove spatial and temporal redundancy. However, efficient exploitation of statistical dependencies measured by a mean squared error (MSE) does not always produce the best psychovisual result. In this paper, we propose a pixel-level texture segmentation approach based on visual relevancy to improve the coding efficiency of newly developed AV1 video codec. Our method performs semantic segmentation and combines regions with similar texture in a video frame. These texture regions are then reconstructed using a motion model at the decoder instead of inter-frame prediction. A Convolutional Neural Networks based semantic segmentation combined with post-processing generates pixel-level texture masks that are more accurate compared to block-based texture masks in our previous work. We show that for many standard test sets, the proposed method achieves significant data rate reductions with improved visual quality.