Rollvox: Real-Time and High-Quality LiDAR Colorization with Rolling Shutter Camera

Abstract

In this study, we propose a novel system for real-time coloring LiDAR point clouds with a low-cost RS camera. The main challenges are dealing with the motion distortion of the RS camera and the multi-sensor time synchronization. To tackle these challenges, we carefully design a hardware synchronizer to ensure the strict alignment of the LiDAR, inertial measurement unit, and RS camera. With accurate timestamps, we first use LiDAR-inertial odometry (LIO) for pose estimation, and the poses of image line exposure are calculated by forward propagation based on a constant velocity motion model. Then, we propose our method based on the RS constraint for colorizing the LiDAR point cloud. For comparison, we colorize the LiDAR point cloud with conventional rolling shutter image undistortion. In the real-world tests, The results show that our proposed method produces more accurate and efficient colorization of point clouds. Besides, considering the situation of readout time not being provided, we propose a method to calibrate the readout time by minimizing the reprojection error of LIO's inter-frame pose and image optical flows. We release our code and self-collected datasets on Github33https://github.com/sheng00125/Rollvox to benefit the community.