Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning

Sen Yang; Leyang Cui; Deng Cai; Xinting Huang; Shuming Shi; Wai Lam

EMNLP 2024finding3 citations

Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning

Sen Yang, Leyang Cui, Deng Cai, Xinting Huang, Shuming Shi, Wai Lam

Abstract

Iterative preference learning, though yielding superior performances, requires online annotated preference labels. In this work, we study strategies to save annotation budgets while achieving competitive or even better performances for iterative preference learning. Built on intuitions from active learning, we empirically show that annotating those response pairs with small margins is generally better than large or random. Besides, experiments under the multi-iteration scenario suggest allocating more annotation budgets in the earlier iterations rather than later ones.

BibTeX

@inproceedings{yang-etal-2024-preference,
    title = "Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning",
    author = "Yang, Sen  and
      Cui, Leyang  and
      Cai, Deng  and
      Huang, Xinting  and
      Shi, Shuming  and
      Lam, Wai",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.382/",
    doi = "10.18653/v1/2024.findings-emnlp.382",
    pages = "6549--6561"
}