A Pre-trained Plug-in Mixture-of-LoRAs Model for Transferable Sequential Recommendation

Abstract

The goal of transferable sequential recommendation (TSR) is to improve the performance of sequential recommenders in multiple target domains leveraging knowledge transferred from source domains. Most existing transferable sequential recommenders rely on item modality information but pay insufficient attention to the significant differences across domains (i.e., distinctive user behavior characteristics) in recommender systems. Besides, they are typically built on specific pre-trained model structures, limiting their model generalization to various different downstream domains and sequential recommenders in practical applications. This paper aims to address these issues by exploring the potential of mixture-of-LoRAs based domain adaption and an ensemble of sequential recommenders for learning generalizable and transferable domain-adaptive item representations. We propose MoL-TSR, a pre-trained plug-in mixture of low-rank adaption model for TSR. Specifically, we first devise a mixture-ofLoRAs based domain projector to produce domain-adaptive item representations based on both item modality and user behavior information for recommendation, enhancing the item discrimination and domain transfer ability. Then, we pre-train an ensemble of sequential recommendation models with contrastive learning to enhance the model-aspect generalization. As a general pretrained model, the mixture-of-LoRAs based domain projector in our MoL-TSR is a plug-in item semantic representation learning module, which is convenient to migrate into other transferable sequential recommenders to enhance item semantics better suited for recommendation. Extensive experiments demonstrate the effectiveness and model generalization of MoL-TSR, promising a practical solution for learning generalizable and transferable item semantic representations in recommendation. Our code is available at https://github.com/RUCAIBox/MoL-TSR.