Enhancing Large Language Models on Domain-specific Tasks: A Novel Training Strategy via Domain Adaptation and Preference Alignment

Abstract

In handling complex, domain-specific tasks, particularly in the context of state-owned assets and enterprises (SOAEs), general LLMs suffer from the knowledge gap due to insufficient exposure to domain-specific corpora, and the value disagreement, as they are aligned with universal values rather than domain-specific ones. To tackle these challenges, we propose a novel training strategy tailored for the SOAEs domain. This strategy includes a improved domain-adaptive pretraining (DAP) phase with a replay mechanism to mitigate catastrophic forgetting. Following DAP, we utilize a selective portion of domain-specific data for supervised fine-tuning (SFT), and innovatively integrate low-quality data with the remaining SFT data to curate tailored preference datasets, leveraging the Kahneman-Tversky Optimization technique to align our LLMs. Our proposed approach effectively utilizes the data that is often discarded in conventional training procedures, highlighting the substantial improvements in model performance and the importance of training methodologies for domain-specific tasks.