RadLing: Towards Efficient Radiology Report Understanding
Rikhiya Ghosh, Oladimeji Farri, Sanjeev Kumar Karn, Manuela Danu, Ramya Vunikili, Larisa Micu
Abstract
Most natural language tasks in the radiology domain use language models pre-trained on biomedical corpus. There are few pretrained language models trained specifically for radiology, and fewer still that have been trained in a low data setting and gone on to produce comparable results in fine-tuning tasks. We present RadLing, a continuously pretrained language model using ELECTRA-small architecture, trained using over 500K radiology reports that can compete with state-of-the-art results for fine tuning tasks in radiology domain. Our main contribution in this paper is knowledge-aware masking which is an taxonomic knowledge-assisted pre-training task that dynamically masks tokens to inject knowledge during pretraining. In addition, we also introduce an knowledge base-aided vocabulary extension to adapt the general tokenization vocabulary to radiology domain.
BibTeX
@inproceedings{ghosh-etal-2023-radling,
title = "{R}ad{L}ing: Towards Efficient Radiology Report Understanding",
author = "Ghosh, Rikhiya and
Farri, Oladimeji and
Karn, Sanjeev Kumar and
Danu, Manuela and
Vunikili, Ramya and
Micu, Larisa",
editor = "Sitaram, Sunayana and
Beigman Klebanov, Beata and
Williams, Jason D",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-industry.61/",
doi = "10.18653/v1/2023.acl-industry.61",
pages = "640--651"
}