Lifelong LERF: Local 3D Semantic Inventory Monitoring Using FogROS2
Adam Rashid, Chung Min Kim, Justin Kerr, Letian Fu, Kush Hari, Ayah Ahmad, Kaiyuan Chen, Huang Huang
Abstract
Inventory monitoring in homes, factories, and retail stores relies on maintaining data despite objects being swapped, added, removed, or moved. We introduce Lifelong LERF, a method that allows a mobile robot with minimal compute to jointly optimize a dense language and geometric representation of its surroundings. Lifelong LERF maintains this representation over time by detecting semantic changes and selectively updating these regions of the environment, avoiding the need to exhaustively remap. Human users can query inventory by providing natural language queries and receiving a 3D heatmap of potential object locations. To manage the computational load, we use Fog-ROS2, a cloud robotics platform, to offload resource-intensive tasks. Lifelong LERF obtains poses from a monocular RGBD SLAM backend, and uses these poses to progressively optimize a Language Embedded Radiance Field (LERF) for semantic monitoring. Experiments with 3-5 objects arranged on a tabletop and a Turtlebot with a RealSense camera suggest that Lifelong LERF can persistently adapt to changes in objects with up to 91% accuracy.
BibTeX
@inproceedings{icra2024_lifelonglerfloca,
title = {Lifelong LERF: Local 3D Semantic Inventory Monitoring Using FogROS2},
author = {Adam Rashid and Chung Min Kim and Justin Kerr and Letian Fu and Kush Hari and Ayah Ahmad and Kaiyuan Chen and Huang Huang and Marcus Gualtieri and Michael Wang and Christian Juette and Nan Tian and Liu Ren and Ken Goldberg},
booktitle = {ICRA 2024},
year = {2024}
}