MetaAct-RL: Training Language Models for Reasoning Through Meta-Action-Based Reinforcement Learning
Zhiheng Xi, Yuhui Wang, Yiwen Ding, Guanyu Li, Senjie Jin, Shichun Liu, Jixuan Huang, Dingwen Yang
Abstract
Outcome-based reinforcement learning has made notable advances in training language models (LMs) for reasoning. However, without explicit incentives and controls, this paradigm has limitations and instability in eliciting high-quality reasoning trajectories with diverse actions—particularly for models whose pretraining lacked extensive reasoning-related data. To this end, we introduce MetaAct-RL, a new RL framework that frames LMs’ thinking as sequential decision making over meta-actions. In this framework, the model chooses and executes a high-level action at each step—such as forward reasoning, critique, or refinement—to gradually reach the correct answer. To encourage deeper exploration, richer action diversity, and to improve sampling efficiency in the RL optimization process, MetaAct-RL incorporates appropriate length-based reward and regularization, and a key-state restart mechanism. Extensive experiments across six benchmarks show that MetaAct-RL improves reasoning performance by 7.99 on Llama3.2-1B and 7.17 on Llama3.1-8B relative to vanilla RL method. Moreover, on the challenging AIME-2024, our method outperforms the vanilla RL by 7.5 with Qwen2.5-1.5B.
BibTeX
@inproceedings{aaai2026_metaactrltrainin,
title = {MetaAct-RL: Training Language Models for Reasoning Through Meta-Action-Based Reinforcement Learning},
author = {Zhiheng Xi and Yuhui Wang and Yiwen Ding and Guanyu Li and Senjie Jin and Shichun Liu and Jixuan Huang and Dingwen Yang and Jiafu Tang and Boyang Hong and Junjie Ye and Shihan Dou and Ming Zhang and Jian Guan and Wei Wu and Rui Zheng and Tao Gui and Qi Zhang and Xuanjing Huang},
booktitle = {AAAI 2026},
year = {2026}
}