Junzhe Shen

TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation

Generating ultra-long sequences with large language models (LLMs) has become increasingly crucial but remains a highly time-intensive task, particularly for sequences up to 100K tokens. While traditional speculative decoding methods exist, simply extending their generation limits fails to accelerate…