Towards Real-Time, Multi-View Video Stereopsis

Abstract

We present a real-time, multi-view video stereopsis (RTMVS) algorithm. This algorithm processes five synchronized video streams from cameras of a stationary camera array using a commodity laptop computer equipped with an Nvidia GPU. It provides 3D visualization of a dynamic scene from a chosen viewpoint at the video frame rate. In RTMVS, 3D surfaces are represented as a set of triangles anchored on a sparse set of 3D feature points. The computationally intensive Structure-from-Motion (SfM) algorithm is executed as an initial step. Feature points in each video stream are tracked using a KLT tracker. Triangles will be updated only when at least one vertex moves from its current position. The algorithm will redetect features every X frame. Epipolar geometry and Trifocal tensor are also exploited to accelerate sparse feature point matching and track filtering. Compared to a dense point cloud multi-view stereopsis baseline algorithm, RTMVS reduces the processing time per frame from 30s down to less than 44 ms.