Associated Learning: an Alternative to End-to-End Backpropagation that Works on CNN, RNN, and Transformer

Dennis Y.H. Wu; Dinan Lin; Vincent Chen; Hung-Hsuan Chen

ICLR 2022poster10 citations

Associated Learning: an Alternative to End-to-End Backpropagation that Works on CNN, RNN, and Transformer

Dennis Y.H. Wu, Dinan Lin, Vincent Chen, Hung-Hsuan Chen

Abstract

This paper studies Associate Learning (AL), an alternative methodology to the end-to-end backpropagation (BP). We introduce the workflow to convert a neural network into a proper structure such that AL can be used to learn the weights for various types of neural networks. We compared AL and BP on some of the most successful types of neural networks -- Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Transformer. Experimental results show that AL consistently outperforms BP on various open datasets. We discuss possible reasons for AL's success and its limitations.

pipeline trainingparallel trainingbackpropagationassociated learning

BibTeX

@inproceedings{
wu2022associated,
title={Associated Learning: an Alternative to End-to-End Backpropagation that Works on {CNN}, {RNN}, and Transformer},
author={Dennis Y.H. Wu and Dinan Lin and Vincent Chen and Hung-Hsuan Chen},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=4N-17dske79}
}