Testing the limits of logical reasoning in neural and hybrid models

Manuel Vargas Guzmán; Jakub Szymanik; Maciej Malicki

NAACL 2024findings3 citations

Testing the limits of logical reasoning in neural and hybrid models

Manuel Vargas Guzmán, Jakub Szymanik, Maciej Malicki

Abstract

We study the ability of neural and hybrid models to generalize logical reasoning patterns. We created a series of tests for analyzing various aspects of generalization in the context of language and reasoning, focusing on compositionality and recursiveness. We used them to study the syllogistic logic in hybrid models, where the network assists in premise selection. We analyzed feed-forward, recurrent, convolutional, and transformer architectures. Our experiments demonstrate that even though the models can capture elementary aspects of the meaning of logical terms, they learn to generalize logical reasoning only to a limited degree.

BibTeX

@inproceedings{guzman-etal-2024-testing,
    title = "Testing the limits of logical reasoning in neural and hybrid models",
    author = "Vargas Guzm{\'a}n, Manuel  and
      Szymanik, Jakub  and
      Malicki, Maciej",
    editor = "Duh, Kevin  and
      Gomez, Helena  and
      Bethard, Steven",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-naacl.147/",
    doi = "10.18653/v1/2024.findings-naacl.147",
    pages = "2267--2279"
}