Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to-SQL
Bing Wang, Yan Gao, Zhoujun Li, Jian-Guang Lou
Abstract
The task of text-to-SQL aims to convert a natural language question into its corresponding SQL query within the context of relational tables. Existing text-to-SQL parsers generate a plausible SQL query for an arbitrary user question, thereby failing to correctly handle problematic user questions. To formalize this problem, we conduct a preliminary study on the observed ambiguous and unanswerable cases in text-to-SQL and summarize them into 6 feature categories. Correspondingly, we identify the causes behind each category and propose requirements for handling ambiguous and unanswerable questions. Following this study, we propose a simple yet effective counterfactual example generation approach that automatically produces ambiguous and unanswerable text-to-SQL examples. Furthermore, we propose a weakly supervised DTE (Detecting-Then-Explaining) model for error detection, localization, and explanation. Experimental results show that our model achieves the best result on both real-world examples and generated examples compared with various baselines. We release our data and code at: https://github.com/wbbeyourself/DTE.
BibTeX
@inproceedings{wang-etal-2023-know,
title = "Know What {I} don`t Know: Handling Ambiguous and Unknown Questions for Text-to-{SQL}",
author = "Wang, Bing and
Gao, Yan and
Li, Zhoujun and
Lou, Jian-Guang",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.findings-acl.352/",
doi = "10.18653/v1/2023.findings-acl.352",
pages = "5701--5714"
}