EMNLP 20250 citations
Representation-based Broad Hallucination Detectors Fail to Generalize Out of Distribution
Zuzanna Dubanowska, Maciej {\.Z}elaszczyk, Micha{\l} Brzozowski, Paolo Mandica, Michal P. Karpowicz
Abstract
We critically assess the efficacy of the current SOTA in hallucination detection and find that its performance on the RAGTruth dataset is largely driven by a spurious correlation with data. Controlling for this effect, state-of-the-art performs no better than supervised linear probes, while requiring extensive hyperparameter tuning across datasets. Out-of-distribution generalization is currently out of reach, with all of the analyzed methods performing close to random. We propose a set of guidelines for hallucination detection and its evaluation.
BibTeX
@inproceedings{emnlp2025_representationba,
title = {Representation-based Broad Hallucination Detectors Fail to Generalize Out of Distribution},
author = {Zuzanna Dubanowska and Maciej {\.Z}elaszczyk and Micha{\l} Brzozowski and Paolo Mandica and Michal P. Karpowicz},
booktitle = {EMNLP 2025},
year = {2025}
}