Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

Taeuk Kim; Jihun Choi; Daniel Edmiston; Sang-goo Lee

ICLR 2020poster115 citations

Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

Taeuk Kim, Jihun Choi, Daniel Edmiston, Sang-goo Lee

Abstract

With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.

BibTeX

@inproceedings{
Kim2020Are,
title={Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction},
author={Taeuk Kim and Jihun Choi and Daniel Edmiston and Sang-goo Lee},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=H1xPR3NtPB}
}