2025
Hymba: A Hybrid-head Architecture for Small Language Models
ICLR 2025spotlight
We propose Hymba, a family of small language models featuring a hybrid-head parallel architecture that integrates attention mechanisms and state space models (SSMs) within the same layer, offering parallel and complementary processing of the same inputs. In this hybrid-head module, attention heads p…