Feiye Huo

RepSpec: Structural Re-parameterized Draft Model Training for Speculative Decoding

As the parameter size of large language models (LLMs) continues to grow, the latency of autoregressive inference increases due to memory-bound computational inefficiency. To address this, speculative decoding has been proposed, where a large target model verifies multiple tokens generated in paralle…