2023
Simpler neural networks prefer subregular languages
EMNLP 2023long findings
We apply a continuous relaxation of $L_0$ regularization (Louizos et al., 2017), which induces sparsity, to study the inductive biases of LSTMs. In particular, we are interested in the patterns of formal languages which are readily learned and expressed by LSTMs. Across a wide range of tests we find…