Token-level Proximal Policy Optimization for Query Generation
Yichen Ouyang, Lu Wang, Fangkai Yang, Pu Zhao, Chenghua Huang, Jianfeng Liu, Bochen Pang, Yaming Yang
Abstract
Query generation is a critical task for web search engines (e.g. Google, Bing) and recommendation systems. Recently, state-of-the-art query generation methods leverage Large Language Models (LLMs) for their strong capabilities in context understanding and text generation. However, they still face challenges in generating high-quality queries in terms of inferring user intent based on their web search interaction history. In this paper, we propose Token-level Proximal Policy Optimization (TPPO), a noval approach designed to empower LLMs perform better in query generation through fine-tuning. TPPO is based on the Reinforcement Learning from AI Feedback (RLAIF) paradigm, consisting of a token-level reward model and a token-level proximal policy optimization module to address the sparse reward challenge in traditional RLAIF frameworks. We conducted experiments on both open-source dataset and an industrial dataset that was collected from a globally-used search engine, demonstrating that TPPO significantly improves the performance of query generation for LLMs and outperforms its existing competitors.
BibTeX
@inproceedings{emnlp2025_tokenlevelproxim,
title = {Token-level Proximal Policy Optimization for Query Generation},
author = {Yichen Ouyang and Lu Wang and Fangkai Yang and Pu Zhao and Chenghua Huang and Jianfeng Liu and Bochen Pang and Yaming Yang and Yuefeng Zhan and Hao Sun and Qingwei Lin and Saravan Rajmohan and Weiwei Deng and Dongmei Zhang and Feng Sun},
booktitle = {EMNLP 2025},
year = {2025}
}