Heterogeneous Sensor Fusion and Active Perception for Transparent Object Reconstruction with a PDM2 Sensor and a Camera

Fengzhi Guo; Shuangyu Xie; Di Wang; Cheng Fang; Jun Zou; Dezhen Song

ICRA 20252 citations

Heterogeneous Sensor Fusion and Active Perception for Transparent Object Reconstruction with a PDM2 Sensor and a Camera

Fengzhi Guo, Shuangyu Xie, Di Wang, Cheng Fang, Jun Zou, Dezhen Song

Abstract

Transparent household objects present a challenge for domestic service robots, since neither regular cameras nor RGB-D cameras can provide accurate points for shape reconstruction. The new type of pretouch dual-modality distance and material sensor (PDM2) can provide reliable and accurate depth readings, but it is a point sensor and scanning the object exclusively with the sensor is too inefficient. Hence, we present a sensor fusion approach by combining a regular camera with the PDM2 sensor. The approach is based on a data fusion algorithm for shape reconstruction and an active perception algorithm for scan planning for the PDM2 sensor. The data fusion algorithm is a distributed Gaussian process (GP)-based shape reconstruction method that allows for incremental local update to reduce computational time. The active perception algorithm is an optimization-based approach by increasing the information gain (IG) and prioritizing the boundary points under a preset travel distance constraint. We have implemented and tested the algorithms with six different transparent household items. The results show satisfactory shape reconstruction results in all test cases with an average increase in intersection over union (IoU) from 0.73 to 0.96.

BibTeX

@inproceedings{icra2025_heterogeneoussen,
  title = {Heterogeneous Sensor Fusion and Active Perception for Transparent Object Reconstruction with a PDM2 Sensor and a Camera},
  author = {Fengzhi Guo and Shuangyu Xie and Di Wang and Cheng Fang and Jun Zou and Dezhen Song},
  booktitle = {ICRA 2025},
  year = {2025}
}