Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition
Floris Hengst, Ralf Wolter, Patrick Altmeyer, Arda Kaygan
Abstract
We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level.By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection.In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection.CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.
BibTeX
@inproceedings{hengst-etal-2024-conformal,
title = "Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition",
author = "Hengst, Floris and
Wolter, Ralf and
Altmeyer, Patrick and
Kaygan, Arda",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-naacl.156/",
doi = "10.18653/v1/2024.findings-naacl.156",
pages = "2412--2432"
}