Bootstrap Predictive Coding: Investigating a Non-Contrastive Self-Supervised Learning Approach

Abstract

Self-supervised learning methods (SSL) have seen wide popularity for speech representation learning. Early methods, such as wav2vec, were causal, whilst more recent approaches, notably wav2vec 2.0 and data2vec, have employed masking strategies together with a Transformer architecture. Many SSL methods use contrastive learning; however, non-contrastive methods, while susceptible to representational collapse, have recently seen success in other fields and have been successfully applied to speech in data2vec. This work returns to the study of causal models, comparing them with equivalent non-causal variants, motivated by our observation that non-contrastive SSL models have never been investigated in this setting. To this end, we propose a novel approach, Bootstrapped Predictive Coding (BPC), a causal non-contrastive SSL model. We find that causal SSL models outperform their non-causal counterparts in both contrastive and non-contrastive training setups, and that representations obtained with BPC give overall best performance when evaluated on phone frame classification.