Multi-Architecture Multi-Expert Diffusion Models
Yunsung Lee, JinYoung Kim, Hyojun Go, Myeongho Jeong, Shinhyeok Oh, Seungtaek Choi
Abstract
In this paper, we address the performance degradation of efficient diffusion models by introducing Multi-architecturE Multi-Expert diffusion models (MEME). We identify the need for tailored operations at different time-steps in diffusion processes and leverage this insight to create compact yet high-performing models. MEME assigns distinct architectures to different time-step intervals, balancing convolution and self-attention operations based on observed frequency characteristics. We also introduce a soft interval assignment strategy for comprehensive training. Empirically, MEME operates 3.3 times faster than baselines while improving image generation quality (FID scores) by 0.62 (FFHQ) and 0.37 (CelebA). Though we validate the effectiveness of assigning more optimal architecture per time-step, where efficient models outperform the larger models, we argue that MEME opens a new design choice for diffusion models that can be easily applied in other scenarios, such as large multi-expert models.
BibTeX
@article{Lee_Kim_Go_Jeong_Oh_Choi_2024, title={Multi-Architecture Multi-Expert Diffusion Models}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/29245}, DOI={10.1609/aaai.v38i12.29245}, abstractNote={In this paper, we address the performance degradation of efficient diffusion models by introducing Multi-architecturE Multi-Expert diffusion models (MEME). We identify the need for tailored operations at different time-steps in diffusion processes and leverage this insight to create compact yet high-performing models. MEME assigns distinct architectures to different time-step intervals, balancing convolution and self-attention operations based on observed frequency characteristics. We also introduce a soft interval assignment strategy for comprehensive training. Empirically, MEME operates 3.3 times faster than baselines while improving image generation quality (FID scores) by 0.62 (FFHQ) and 0.37 (CelebA). Though we validate the effectiveness of assigning more optimal architecture per time-step, where efficient models outperform the larger models, we argue that MEME opens a new design choice for diffusion models that can be easily applied in other scenarios, such as large multi-expert models.}, number={12}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Lee, Yunsung and Kim, JinYoung and Go, Hyojun and Jeong, Myeongho and Oh, Shinhyeok and Choi, Seungtaek}, year={2024}, month={Mar.}, pages={13427-13436} }