2025
ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models
COLING 2025main
Activation sparsity refers to the existence of considerable weakly-contributed elements among activation outputs, serving as a promising paradigm for accelerating model inference. Nevertheless, most large language models (LLMs) adopt activation functions without intrinsic activation sparsity (e.g.,…