Real-Time Light Field Processing for Autonomous Robotics

Abstract

Typical autonomous robotics systems incorporate multiple cameras, LIDAR sensors and sophisticated computing resources. In this paper we present a software framework for utilizing any array of multiple cameras with sufficient field-of-view (FOV) overlap as a light field imaging system. We show that the typical linear arrays that exist on autonomous cars are sufficient to capture stable time resolved light fields even when moving at highway speeds. We elaborate on the potential pitfalls associated with such a technique namely loss of calibration between cameras due to high frequency vibrations and sudden shocks associated with driving over potholes and highlight a method that can compensate for such effects. We demonstrate that the light fields collected by simple linear arrays can be processed in real time for a wide variety of useful applications including occlusion removal, for signal enhancement in featureless images captured in very low light, for reflection removal and for improved visibility in extreme conditions associated with snow and heavy rain.