P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering
Wenlin Zhong, Chengyuan Liu, Yiquan Wu, Bovin Tan, Changlong Sun, Yi Wang, Xiaozhong Liu, Kun Kuang
Abstract
While reinforcement learning with verifiable rewards (RLVR) has advanced LLM reasoning in structured domains like mathematics and programming, its application to general-domain reasoning tasks remains challenging due to the absence of verifiable reward signals. To this end, methods like Reinforcement Learning with Reference Probability Reward (RLPR) have emerged, leveraging the probability of generating the final answer as a reward signal. However, these outcome-focused approaches neglect crucial step-by-step supervision of the reasoning process itself. To address this gap, we introduce Probabilistic Process Supervision (P2S), a novel self-supervision framework that provides fine-grained process rewards without requiring a separate reward model or human-annotated reasoning steps. During reinforcement learning, P2S synthesizes and filters a high-quality reference reasoning chain (gold-CoT). The core of our method is to calculate a Path Faithfulness Reward (PFR) for each reasoning step, which is derived from the conditional probability of generating the gold-CoT
BibTeX
@inproceedings{aaai2026_p2sprobabilistic,
title = {P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering},
author = {Wenlin Zhong and Chengyuan Liu and Yiquan Wu and Bovin Tan and Changlong Sun and Yi Wang and Xiaozhong Liu and Kun Kuang},
booktitle = {AAAI 2026},
year = {2026}
}