← Search

Zhongyisun Sun

2 accepted papers

2024

Anchor-based Robust Finetuning of Vision-Language Models

CVPR 2024poster

We aim at finetuning a vision-language model without hurting its out-of-distribution (OOD) generalization. We address two types of OOD generalization i.e. i) domain shift such as natural to sketch images and ii) zero-shot capability to recognize the category that was not contained in the finetune da…

Cited by 9SourcePDFScholar
2024

Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models

ICML 2024poster

Catastrophic forgetting emerges as a critical challenge when fine-tuning multi-modal large language models (MLLMs), where improving performance on unseen tasks often leads to a significant performance drop on the original tasks. This paper presents a comprehensive analysis of catastrophic forgetting…

Cited by 33SourcePDFScholar