Benchmarking Simulated Robotic Manipulation Through a Real World Dataset
Jack Collins, Jessie McVicar, David Wedlock, Ross Brown, David Howard, Jürgen Leitner
Abstract
We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset. Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset. The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies. We apply our benchmark to two simulation environments, i.e., PyBullet and V-Rep, and publish the results. All materials required to benchmark an environment, including protocols and the dataset, can be found at the benchmarks’ website <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><sup>1</sup></xref> <fn id="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><label><sup>1</sup></label> [Online]. Available: <uri>https://research.csiro.au/robotics/manipulation-benchmark/</uri>. </fn> .
BibTeX
@inproceedings{ral2020_benchmarkingsimu,
title = {Benchmarking Simulated Robotic Manipulation Through a Real World Dataset},
author = {Jack Collins and Jessie McVicar and David Wedlock and Ross Brown and David Howard and Jürgen Leitner},
booktitle = {RA-L 2020},
year = {2020}
}