Karol Pustelnik

Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference

While transformer models have been highly successful, they are computationally inefficient. We observe that for each layer, the full width of the layer may be needed only for a small subset of tokens inside a batch and that the "effective" width needed to process a token can vary from layer to layer…