Exploring Meta Evidence for Prompt Optimization

Abstract

Prompt optimization aims to construct effective prompts to fully leverage the capabilities of large language models (LLMs). The key challenge lies in analyzing the outputs of LLMs, identifying fine-grained errors as explicit evidence and guiding reverse prompt correction. However, existing dominant methods feed all evidence into LLMs, leading to excessive redundancy that misleads the models. Moreover, this redundant input could increase computational costs. To address the issue, we introduce a novel meta evidence-aware prompt optimization method (MEPO), which performs fine-grained analysis of the causal relationships within evidence using clustering and Bayesian inference. Initially, MEPO generates preliminary evidence to describe the drawbacks of the given prompt. Subsequently, the evidence is clustered and constructed into a directed acyclic graph (DAG) to produce meta evidence. Finally, we edit the initial prompt based on the obtained meta evidence and employ the beam search to select the best-performing prompt. Experimental results show that MEPO outperforms state-of-the-art baselines, achieving absolute F1 score improvements of 5.8% and 5.0% on two common-used Liar and Jailbreak datasets, respectively. The datasets and codes are available at https://anonymous.4open.science/r/MEPO-E1A0.