Benjamin Bolte — accepted AI-conference papers

2021

Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?

Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, Abdelrahman Mohamed

ICASSP 2021accepted

Compared to vision and language applications, self-supervised pre-training approaches for ASR are challenged by three unique problems: (1) There are multiple sound units in each input utterance, (2) With audio-only pre-training, there is no lexicon of sound units, and (3) Sound units have variable l…

Cited by 0 Source ↗Scholar ↗

2021

On the Influence of Masking Policies in Intermediate Pre-training

Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih +2

EMNLP 2021main

Current NLP models are predominantly trained through a two-stage “pre-train then fine-tune” pipeline. Prior work has shown that inserting an intermediate pre-training stage, using heuristic masking policies for masked language modeling (MLM), can significantly improve final performance. However, it…

Cited by 12 Source ↗PDF ↗code ↗Scholar ↗