Absolute Localization through Vision Transformer Matching of Planetary Surface Perspective Imagery from a Digital Twin

Abstract

We present a novel machine learning framework and synthetic dataset for performing absolute localization on planetary surfaces where satellite navigation systems are unavailable. Current approaches involve manual surface-to-satellite image matching by human rover operators, limiting the rate of planetary exploration and scientific utilization. Our framework leverages deep neural networks to perform image similarity matching between a rover’s onboard cameras and corresponding ground-view images from a digital twin environment created from satellite and elevation maps. The rover views, satellite, and elevation maps are taken from a photorealistic lunar environment simulated in a 3D graphics engine (Unreal Engine 4). The synthetic twin ground-view re-projections are generated using an open-source 3D graphics software (Blender). In total, we generate a dataset of 1.68 million images at 210,000 locations. The images and corresponding metadata are then used to train a DINOv2 vision transformer image similarity model through supervised fine-tuning to determine matching locations between the rover views and candidate re-projections. Through this method, our model is able to determine the ground truth location within 5 m using just 2.5% of the search space, outperforming other deep learning and classical image comparison benchmarks.