WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens

Jian Yang; Dacheng Yin; Xiaoxuan He; Yong Li; Fengyun Rao; Jing Lyu; Wei Zhai; Yang Cao

CVPR 20260 citations

WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens

Jian Yang, Dacheng Yin, Xiaoxuan He, Yong Li, Fengyun Rao, Jing Lyu, Wei Zhai, Yang Cao

Abstract

Recent progress in multimodal large language models (MLLMs) has highlighted the challenge of efficiently bridging pre-trained Vision-Language Models (VLMs) with Diffusion Models. While methods using a fixed number of learnable query tokens offer computational efficiency, they suffer from task generalization collapse, failing to adapt to new tasks that are distant from their pre-training tasks. To overcome this, we propose Noisy Query Tokens, which learn a distributed representation space between the VLM and Diffusion Model via end-to-end optimization, enhancing continual learning. Additionally, we introduce a VAE branch with linear projection to recover fine-grained image details. Experimental results confirm our approach mitigates generalization collapse and enables stable continual learning across diverse tasks.

BibTeX

@inproceedings{cvpr2026_wemmuenhancedbri,
  title = {WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens},
  author = {Jian Yang and Dacheng Yin and Xiaoxuan He and Yong Li and Fengyun Rao and Jing Lyu and Wei Zhai and Yang Cao and Zheng-Jun Zha},
  booktitle = {CVPR 2026},
  year = {2026}
}