Aurelio Sulser

Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory

Despite their successes, deep learning models struggle with tasks requiring complex reasoning and function composition. We present a theoretical and empirical investigation into the limitations of Structured State Space Models (SSMs) and Transformers in such tasks. We prove that one-layer SSMs canno…