Back Razor: Memory-Efficient Transfer Learning by Self-Sparsified Backpropagation
Transfer learning from the model trained on large datasets to customized downstream tasks has been widely used as the pre-trained model can greatly boost the generalizability. However, the increasing sizes of pre-trained models also lead to a prohibitively large memory footprints for downstream tran…