DIFFUSION TIMBRE TRANSFER VIA MUTUAL INFORMATION GUIDED INPAINTING
Ching Ho Lee, Javier Nistal, Marco Pasini
Abstract
We study timbre transfer as an inference-time editing problem for music audio. Starting from a strong pre-trained latent diffusion model, we introduce a lightweight procedure that requires no additional training: (i) a dimension-wise noise injection that targets latent channels most informative of instrument identity, and (ii) an early-step clamping mechanism that re-imposes the input's melodic and rhythmic structure during reverse diffusion. The method operates directly on audio latents and is compatible with text/audio conditioning (e.g., CLAP). We discuss design choices,analyze trade-offs between timbral change and structural preservation, and show that simple inference-time controls can meaningfully steer pre-trained models for style-transfer use cases.
BibTeX
@inproceedings{icassp2026_diffusiontimbret,
title = {DIFFUSION TIMBRE TRANSFER VIA MUTUAL INFORMATION GUIDED INPAINTING},
author = {Ching Ho Lee and Javier Nistal and Marco Pasini},
booktitle = {ICASSP 2026},
year = {2026}
}