Parameter-Efficient Federal-Tuning Enhances Privacy Preserving for Speech Emotion Recognition

Abstract

The Pre-trained Speech Models (PSMs) generate universal speech representations using self-supervised or weakly-supervised learning from large-scale datasets. It achieves promising performance when fine-tuned for specific tasks such as Speech Emotion Recognition (SER). However, fine-tuning on various datasets requires storing the entire model’s weight parameters, complicating real-world deployment. Additionally, centralized fine-tuning relies on user data, posing significant privacy risks. To address these challenges, we propose employing Federated Learning (FL) for fine-tuning PSMs with a Parameter-Efficient Fine-Tuning (PEFT) method. By embedding trainable layers in the feed-forward layers of the pre-trained model, we keep the backbone model frozen and only update the trainable layer parameters during federated training, significantly reducing parameter transmission. Specifically, we evaluated the performance of downstream model fine-tuning, adapter tuning, embedding prompt tuning, and LoRA within a federated fine-tuning framework for PSMs, demonstrating the framework’s feasibility and effectiveness. Furthermore, attribute inference attack tests showed that gender inference results on three datasets were at chance levels.