Self-Supervised Optical Flow with Spiking Neural Networks and Event Based Cameras
Kenneth Chaney, Artemis Panagopoulou, Chankyu Lee, Kaushik Roy, Kostas Daniilidis
Abstract
Optical flow can be leveraged in robotic systems for obstacle detection where low latency solutions are critical in highly dynamic settings. While event-based cameras have changed the dominant paradigm of sending by encoding stimuli into spike trails, offering low bandwidth and latency, events are still processed with traditional convolutional networks in GPUs defeating, thus, the promise of efficient low capacity low power processing that inspired the design of event sensors. In this work, we introduce a shallow spiking neural network for the computation of optical flow consisting of Leaky Integrate and Fire neurons.Optical flow is predicted as the synthesis of motion orientation selective channels. Learning is accomplished by Back-propapagation Through Time. We present promising results on events recorded in real "in the wild" scenes that has the capability to use only a small fraction of the energy consumed in CNNs deployed on GPUs.
BibTeX
@inproceedings{iros2021_selfsupervisedop,
title = {Self-Supervised Optical Flow with Spiking Neural Networks and Event Based Cameras},
author = {Kenneth Chaney and Artemis Panagopoulou and Chankyu Lee and Kaushik Roy and Kostas Daniilidis},
booktitle = {IROS 2021},
year = {2021}
}