Arnaud Pannatier

HyperMixer: An MLP-based Low Cost Alternative to Transformers

Transformer-based architectures are the model of choice for natural language understanding, but they come at a significant cost, as they have quadratic complexity in the input length, require a lot of training data, and can be difficult to tune. In the pursuit of lower costs, we investigate simple M…