Poisoning The Diffusion: A Simple and Robust Watermarking Method for Audio Generation

Abstract

As the capabilities of diffusion-based audio generation models advance, intellectual property protection for both generated audio and model weights becomes important. However, current audio watermarking methods mainly integrate separate encode and decode modules to watermark the generated audio, which not only imposes unnecessary computational overhead, but also neglects crucial protection of model weights. In this paper, we introduce a novel, simple yet robust watermarking algorithm, termed Poisoning The Diffusion(PTD), tailored for diffusion models. Specifically, PTD incorporates a "poison" vector K during the training phase, modifying the noise distribution in the forward process. Subsequently, the "poisoned" diffusion outputs audio with a unique noise distribution that can be quantitatively identified using numerical testing methods, thus effectively serving as an audio watermark. Furthermore, in the reverse process, in the absence of the "antidote" vector (i.e. authorization), the model is unable to perform correct sampling operations, thereby preventing unauthorized use and fine-tuning. Rigorous experimental validation on datasets comprising vocal and musical compositions confirms the efficacy of PTD, underscoring its stability and robustness.