The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank
Daphne Ippolito, Liam Dugan, Emily Reif, Ann Yuan, Andy Coenen, Chris Callison-Burch
Abstract
The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do fill in the blank, a more useful model is one that can effectively perform _both_ FitB and continuation tasks. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how these models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.
BibTeX
@inproceedings{ippolito-etal-2022-case,
title = "The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank",
author = "Ippolito, Daphne and
Dugan, Liam and
Reif, Emily and
Yuan, Ann and
Coenen, Andy and
Callison-Burch, Chris",
editor = "Carpuat, Marine and
de Marneffe, Marie-Catherine and
Meza Ruiz, Ivan Vladimir",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
month = jul,
year = "2022",
address = "Seattle, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-naacl.185/",
doi = "10.18653/v1/2022.findings-naacl.185",
pages = "2421--2432"
}