Identifying Performance Regression Conditions for Testing & Evaluation of Autonomous Systems
Paul Stankiewicz, Marin Kobilarov
Abstract
This paper addresses the problem of identifying whether/how a black-box autonomous system has regressed in performance when compared to previous versions. The approach analyzes performance datasets (typically gathered through simulation-based testing) and automatically extracts test parameter clusters of predicted performance regression. First, surrogate modeling with quantile random forests is used to predict regions of performance regression with high confidence. The predicted regression landscape is then clustered in both the output space and input space to produce groupings of test conditions ranked by performance regression severity. This approach is analyzed using randomized test functions as well as through a case study to detect performance regression in autonomous surface vessel software.
BibTeX
@inproceedings{iros2021_identifyingperfo,
title = {Identifying Performance Regression Conditions for Testing & Evaluation of Autonomous Systems},
author = {Paul Stankiewicz and Marin Kobilarov},
booktitle = {IROS 2021},
year = {2021}
}