OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations
Christina Kassab, Sacha Morin, Martin Büchner, Matias Mattamala, Kumaraditya Gupta, Abhinav Valada, Liam Paull, Maurice Fallon
Abstract
3D scene understanding has been transformed by open-vocabulary language models that enable interaction via natural language. However, at present the evaluation of these representations is limited to datasets with closed-set semantics that do not capture the richness of language. This work presents OpenLex3D, a dedicated benchmark for evaluating 3D open-vocabulary scene representations. OpenLex3D provides entirely new label annotations for scenes from Replica, ScanNet++, and HM3D, which capture real-world linguistic variability by introducing synonymical object categories and additional nuanced descriptions. Our label sets provide 13 times more labels per scene than the original datasets. By introducing an open-set 3D semantic segmentation task and an object retrieval task, we evaluate various existing 3D open-vocabulary methods on OpenLex3D, showcasing failure cases, and avenues for improvement. Our experiments provide insights on feature precision, segmentation, and downstream capabilities. The benchmark is publicly available at: https://openlex3d.github.io/.
BibTeX
@inproceedings{
kassab2025openlexd,
title={OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations},
author={Christina Kassab and Sacha Morin and Martin B{\"u}chner and Matias Mattamala and Kumaraditya Gupta and Abhinav Valada and Liam Paull and Maurice Fallon},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2025},
url={https://openreview.net/forum?id=XZPGUWOXeP}
}