Do Language Models Understand Measurements?

Sungjin Park; Seungwoo Ryu; Edward Choi

EMNLP 2022finding10 citations

Do Language Models Understand Measurements?

Abstract

Recent success of pre-trained language models (PLMs) has stimulated interest in their ability to understand and work with numbers. Yet, the numerical reasoning over measurements has not been formally studied despite their importance. In this study, we show that PLMs lack the capability required for reasoning over measurements. Furthermore, we find that a language model trained on a measurement-rich corpus shows better performance on understanding measurements. We propose a simple embedding strategy to better distinguish between numbers and units, which leads to a significant improvement in the probing tasks.

BibTeX

@inproceedings{park-etal-2022-language,
    title = "Do Language Models Understand Measurements?",
    author = "Park, Sungjin  and
      Ryu, Seungwoo  and
      Choi, Edward",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.findings-emnlp.128/",
    doi = "10.18653/v1/2022.findings-emnlp.128",
    pages = "1782--1792"
}