Kevin C. Wibisono

Bidirectional Attention as a Mixture of Continuous Word Experts

Bidirectional attention - composed of the neural network architecture of self-attention with positional encodings, together with the masked language model (MLM) objective - has emerged as a key component of modern large language models (LLMs). Despite its empirical success, few studies have examined…