NAACL 2021long6 citations
User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization
Shohei Higashiyama, Masao Utiyama, Taro Watanabe, Eiichiro Sumita
Abstract
Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.
BibTeX
@inproceedings{higashiyama-etal-2021-user,
title = "User-Generated Text Corpus for Evaluating {J}apanese Morphological Analysis and Lexical Normalization",
author = "Higashiyama, Shohei and
Utiyama, Masao and
Watanabe, Taro and
Sumita, Eiichiro",
editor = "Toutanova, Kristina and
Rumshisky, Anna and
Zettlemoyer, Luke and
Hakkani-Tur, Dilek and
Beltagy, Iz and
Bethard, Steven and
Cotterell, Ryan and
Chakraborty, Tanmoy and
Zhou, Yichao",
booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jun,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.naacl-main.438/",
doi = "10.18653/v1/2021.naacl-main.438",
pages = "5532--5541"
}