What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim
Abstract
GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.
BibTeX
@inproceedings{kim-etal-2021-changes,
title = "What Changes Can Large-scale Language Models Bring? Intensive Study on {H}yper{CLOVA}: Billions-scale {K}orean Generative Pretrained Transformers",
author = "Kim, Boseop and
Kim, HyoungSeok and
Lee, Sang-Woo and
Lee, Gichang and
Kwak, Donghyun and
Dong Hyeon, Jeon and
Park, Sunghyun and
Kim, Sungju and
Kim, Seonhoon and
Seo, Dongpil and
Lee, Heungsub and
Jeong, Minyoung and
Lee, Sungjae and
Kim, Minsub and
Ko, Suk Hyun and
Kim, Seokhun and
Park, Taeyong and
Kim, Jinuk and
Kang, Soyoung and
Ryu, Na-Hyeon and
Yoo, Kang Min and
Chang, Minsuk and
Suh, Soobin and
In, Sookyo and
Park, Jinseong and
Kim, Kyungduk and
Kim, Hiun and
Jeong, Jisu and
Yeo, Yong Goo and
Ham, Donghoon and
Park, Dongju and
Lee, Min Young and
Kang, Jaewook and
Kang, Inho and
Ha, Jung-Woo and
Park, Woomyoung and
Sung, Nako",
editor = "Moens, Marie-Francine and
Huang, Xuanjing and
Specia, Lucia and
Yih, Scott Wen-tau",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.274/",
doi = "10.18653/v1/2021.emnlp-main.274",
pages = "3405--3424"
}