Self-Evaluation of Large Language Model based on Glass-box Features
Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, Tiejun Zhao, Wenpeng Lu
Abstract
The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect – model-aware glass-box features – is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable quality indicator for self-evaluation. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features.
BibTeX
@inproceedings{huang-etal-2024-self,
title = "Self-Evaluation of Large Language Model based on Glass-box Features",
author = "Huang, Hui and
Qu, Yingqi and
Liu, Jing and
Yang, Muyun and
Xu, Bing and
Zhao, Tiejun and
Lu, Wenpeng",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.333/",
doi = "10.18653/v1/2024.findings-emnlp.333",
pages = "5813--5820"
}