Beichen Xue

State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory

State-space models have gained popularity in sequence modelling due to their simple and efficient network structures. However, the absence of nonlinear activation along the temporal direction limits the model's capacity. In this paper, we prove that stacking state-space models with layer-wise nonlin…