Fine-Tuning Language Models with Collaborative and Semantic Experts
Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Lei Zhang, Qiang Qu, Junyang Lin
Abstract
Recent advancements in large language models (LLMs) have broadened their application scope but revealed challenges in balancing capabilities across general knowledge, coding, and mathematics. To address this, we introduce a Collaborative and Semantic Experts (CoE) approach for supervised fine-tuning (SFT), which employs a two-phase training strategy. Initially, expert training fine-tunes the feed-forward network on specialized datasets, developing distinct experts in targeted domains. Subsequently, expert leveraging synthesizes these trained experts into a structured model with semantic guidance to activate specific experts, enhancing performance and interpretability. Evaluations on comprehensive benchmarks across MMLU, HumanEval, GSM8K, MT-Bench, and AlpacaEval confirm CoE's efficacy, demonstrating improved performance and expert collaboration in diverse tasks, significantly outperforming traditional SFT methods.
BibTeX
@article{Yang_Hui_Yang_Yang_Zhang_Qu_Lin_2025, title={Fine-Tuning Language Models with Collaborative and Semantic Experts}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/34753}, DOI={10.1609/aaai.v39i24.34753}, abstractNote={Recent advancements in large language models (LLMs) have broadened their application scope but revealed challenges in balancing capabilities across general knowledge, coding, and mathematics. To address this, we introduce a Collaborative and Semantic Experts (CoE) approach for supervised fine-tuning (SFT), which employs a two-phase training strategy. Initially, expert training fine-tunes the feed-forward network on specialized datasets, developing distinct experts in targeted domains. Subsequently, expert leveraging synthesizes these trained experts into a structured model with semantic guidance to activate specific experts, enhancing performance and interpretability. Evaluations on comprehensive benchmarks across MMLU, HumanEval, GSM8K, MT-Bench, and AlpacaEval confirm CoE’s efficacy, demonstrating improved performance and expert collaboration in diverse tasks, significantly outperforming traditional SFT methods.}, number={24}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Yang, Jiaxi and Hui, Binyuan and Yang, Min and Yang, Jian and Zhang, Lei and Qu, Qiang and Lin, Junyang}, year={2025}, month={Apr.}, pages={25624-25632} }