The Impact of Language on Arithmetic Proficiency: A Multilingual Investigation with Cross-Agent Checking Computation
Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao
Abstract
This paper critically examines the arithmetic capabilities of Large Language Models (LLMs), uncovering significant limitations in their performance. Our research reveals a notable decline in accuracy for complex calculations involving large numbers, with addition and subtraction tasks showing varying degrees of proficiency. Additionally, we challenge the notion that arithmetic is language-independent, finding up to a 10% difference in performance across twenty languages. The study also compares self-verification methods with cross-agent collaborations, showing that a single model often outperforms collaborative approaches in basic arithmetic tasks. These findings suggest a need to reassess the effectiveness of LLMs in tasks requiring numerical accuracy and precision.
BibTeX
@inproceedings{chen-etal-2024-impact,
title = "The Impact of Language on Arithmetic Proficiency: A Multilingual Investigation with Cross-Agent Checking Computation",
author = "Chen, Chung-Chi and
Takamura, Hiroya and
Kobayashi, Ichiro and
Miyao, Yusuke",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.naacl-short.53/",
doi = "10.18653/v1/2024.naacl-short.53",
pages = "631--637"
}