Maximilian Holsman

Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff

Speculative Decoding (SD) enforces strict distributional equivalence to the target model when accepting candidate tokens. While it maintains the target model’s generation quality, this strict equivalence limits the speedup achievable by SD and prevents users from trading deviations from the target d…