Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning

Andrew Keenan Richardson; Ryan Othniel Kearns; Sean Moss; Vincent Wang; Philipp Koralus

ICLR 2026poster0 citations

Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning

Andrew Keenan Richardson, Ryan Othniel Kearns, Sean Moss, Vincent Wang, Philipp Koralus

Abstract

We study logical reasoning in language models by asking whether their errors follow established human fallacy patterns. Using the Erotetic Theory of Reasoning (ETR) and its open‑source implementation, PyETR, we programmatically generate 383 formally specified reasoning problems and evaluate 38 models. For each response, we judge logical correctness and, when incorrect, whether it matches an ETR‑predicted fallacy. Two results stand out: (i) as a capability proxy (Chatbot Arena Elo) increases, a larger share of a model’s incorrect answers are ETR‑predicted fallacies ($\rho=0.360, p=0.0265$), while overall correctness on this dataset shows no correlation with capability; (ii) reversing premise order significantly reduces fallacy production for many models, mirroring human order effects. Methodologically, PyETR provides an open‑source pipeline for unbounded, synthetic, contamination‑resistant reasoning tests linked to a cognitive theory, enabling analyses that focus on error composition rather than error rate.

Language modelsreasoningsynthetic datacontamination-proofhuman-like errorscognitive fallaciesErotetic Theory of ReasoningPyETRlogical fallacieshuman-like reasoning patternsreasoning evaluationquestion-driven inferenceinverse scaling lawshuman cognitionrationality vs fallibilitycognitive biasesorder effectsreasoning benchmarkscognitive science alignmentAI alignmentsystematic deviations from logicnormative vs descriptive reasoningreasoning tasksdisjunction fallacymodus ponensmodus tollenssyllogistic inferencelogical validitydata contaminationnatural language reasoning tasksformal semanticsmental modelsevaluation harnessChatbot Arenamedical diagnosislegal reasoninghigh-stakes decision-makingalignment benchmarksrobust reasoning systemsAI evaluation frameworks

BibTeX

@inproceedings{
richardson2026theorygrounded,
title={Theory-Grounded Evaluation of Human-Like Fallacy Patterns in {LLM} Reasoning},
author={Andrew Keenan Richardson and Ryan Othniel Kearns and Sean Moss and Vincent Wang and Philipp Koralus},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=1HjzhdTEC7}
}