Training Medical QA Models Based on Mixed Rewards from Multiple-Choice and Open-Ended Questions
Yue Qiu, Yujan Ting, Pei Dong, Terrence Chen, Weijing Huang
Abstract
Reinforcement learning (RL) for large language models (LLMs) typically requires clear reward signals, which are often unavailable for open-ended (OE) questions where answer evaluation is ambiguous without scalable expert labeling. We investigate whether LLMs benefit from training on mixed data with varying reward clarity. Our approach combines Multiple-choice questions (MCQs), which offer clear binary rewards, with OE questions, for which we use simpler, potentially noisy rewards such as Jaccard similarity or LLM-based evaluators. We hypothesize that MCQs can stabilize training when mixed with OE questions. Our experiments show this mixed-data approach consistently improves medical question-answering performance across model scales.
BibTeX
@inproceedings{emnlp2025_trainingmedicalq,
title = {Training Medical QA Models Based on Mixed Rewards from Multiple-Choice and Open-Ended Questions},
author = {Yue Qiu and Yujan Ting and Pei Dong and Terrence Chen and Weijing Huang},
booktitle = {EMNLP 2025},
year = {2025}
}