Łukasz Gniecki

Sparser, Better, Deeper, Stronger: Improving Static Sparse Training with Exact Orthogonal Initialization

Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. A key design choice is given by the sparse initialization, which determines the trainable sub-network through a binary mask. Existing methods mainly select such mask based on a predefined d…