2022
Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction
NeurIPS 2022accept
Training deep learning models can be computationally expensive. Prior works have shown that increasing the batch size can potentially lead to better overall throughput. However, the batch size is frequently limited by the accelerator memory capacity due to the activations/feature maps stored for the…