DePO: Demonstration-guided Policy Optimization for Molecular Optimization
Xuan Li, Zhanke Zhou, Zongze Li, Jiangchao Yao, Yu Rong, Lu Zhang, Bo Han
Abstract
Large language models (LLMs) exhibit remarkable mathematical reasoning abilities through supervised fine-tuning (SFT) or reinforcement learning with verifiable rewards (RLVR). However, adapting them to scientific domains like molecular optimization is challenging: its datasets provide only reference molecules, lacking the reasoning traces for SFT, while its competitive objectives hinder RLVR. To address these issues, we introduce Demonstration-guided Policy Optimization (DePO). We leverage reference molecules as supervised signals to regularize the search direction while preserving the model’s reasoning capabilities. Experiments show that DePO significantly outperforms both SFT and RLVR across key molecular optimization metrics, and excels in balancing the competitive optimization objectives. DePO achieves up to 13\% improvement compared to SFT and other baseline approaches. DePO also shows generalization capabilities and inference-scaling properties.
BibTeX
@inproceedings{
li2026depo,
title={De{PO}: Demonstration-guided Policy Optimization for Molecular Optimization},
author={Xuan Li and Zhanke Zhou and Zongze Li and Jiangchao Yao and Yu Rong and Lu Zhang and Bo Han},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=m4nvqQkm4X}
}