Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models

Elena Stringli; Maria Lymperaiou; Giorgos Filandrianos; Athanasios Voulodimos; Giorgos Stamou

ACL 2025finding0 citations

Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models

Elena Stringli, Maria Lymperaiou, Giorgos Filandrianos, Athanasios Voulodimos, Giorgos Stamou

Abstract

Inverse tasks can uncover potential reasoning gaps as Large Language Models (LLMs) scale up. In this work, we explore the redefinition task, in which we assign alternative values to well-known physical constants and units of measure, prompting LLMs to respond accordingly. Our findings show that not only does model performance degrade with scale, but its false confidence also rises. Moreover, while factors such as prompting strategies or response formatting are influential, they do not preclude LLMs from anchoring to memorized values.

BibTeX

@inproceedings{stringli-etal-2025-pitfalls,
    title = "Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models",
    author = "Stringli, Elena  and
      Lymperaiou, Maria  and
      Filandrianos, Giorgos  and
      Voulodimos, Athanasios  and
      Stamou, Giorgos",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.492/",
    doi = "10.18653/v1/2025.findings-acl.492",
    pages = "9445--9469",
    ISBN = "979-8-89176-256-5"
}