Opher Lieber — accepted AI-conference papers

2025

Jamba: Hybrid Transformer-Mamba Language Models

Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg +55

ICLR 2025poster

We present Jamba, a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Jamba interleaves blocks of Transformer and Mamba layers, enjoying the benefits of both model families. MoE is added in some of these layers to increase model capacity while keeping active parameter usage manag…

Cited by 3 Source ↗PDF ↗Scholar ↗

2021

PMI-Masking: Principled masking of correlated spans

Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown, Moshe Tennenholtz +1

ICLR 2021spotlight

Masking tokens uniformly at random constitutes a common flaw in the pretraining of Masked Language Models (MLMs) such as BERT. We show that such uniform masking allows an MLM to minimize its training objective by latching onto shallow local signals, leading to pretraining inefficiency and suboptimal…

Cited by 81 Source ↗PDF ↗Scholar ↗