Lawrence Cavedon

Evaluating Numeracy of Language Models as a Natural Language Inference Task

While recent advancements in large language models (LLMs) have enhanced their capabilities to solve mathematical problems, other aspects of numeracy remain underexplored. In this paper, we propose a benchmark to evaluate the ability of language models to perform basic numeracy tasks. We frame numera…