MLPerf Tiny Benchmark
Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Nat Jeffries, Csaba Kiraly, Jeremy Holleman, Pietro Montino, David Kanter
Abstract
Advancements in ultra-low-power tiny machine learning (TinyML) systems promise to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted and easily reproducible benchmark for these systems. To meet this need, we present MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems. The benchmark suite is the collaborative effort of more than 50 organizations from industry and academia and reflects the needs of the community. MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems. Additionally, MLPerf Tiny implements a modular design that enables benchmark submitters to show the benefits of their product, regardless of where it falls on the ML deployment stack, in a fair and reproducible manner. The suite features four benchmarks: keyword spotting, visual wake words, image classification, and anomaly detection.
BibTeX
@inproceedings{
banbury2021mlperf,
title={{MLP}erf Tiny Benchmark},
author={Colby Banbury and Vijay Janapa Reddi and Peter Torelli and Nat Jeffries and Csaba Kiraly and Jeremy Holleman and Pietro Montino and David Kanter and Pete Warden and Danilo Pau and Urmish Thakker and antonio torrini and jay cordaro and Giuseppe Di Guglielmo and Javier Duarte and Honson Tran and Nhan Tran and niu wenxu and xu xuesong},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)},
year={2021},
url={https://openreview.net/forum?id=8RxxwAut1BI}
}