Zayd Muhammad Kawakibi Zuhri — accepted AI-conference papers

2026

Predicting the Order of Upcoming Tokens Improves Language Modeling

Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi, Alham Fikri Aji

ICML 2026poster

Multi-token prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, underperforming in standard NLP benchmarks. We found MTP's exact future token prediction to be too difficult as an auxiliary…

Cited by 0 Source ↗Scholar ↗

2025

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding

Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, Alham Fikri Aji

NAACL 2025findings

Auto-regressive inference of transformers benefit greatly from Key-Value (KV) caching, but can lead to major memory bottlenecks as model size, batch size, and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV) sharing, a novel approach extending KV sharing across transformer la…

Cited by 13 Source ↗PDF ↗code ↗Scholar ↗