Michael Tschannen*

GIVT: Generative Infinite-Vocabulary Transformers

"We introduce Generative Infinite-Vocabulary Transformers(GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the f…