Jaewon Cheon

COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projection

The growing size of large language models has created significant computational inefficiencies. To address this challenge, sparse activation selectively deactivates non-essential parameters during inference, reducing computational costs in FFNN layers. While existing methods focus on non-linear gati