GSMM: Efficient Global Sparsification for Resource-Conscious Multimodal Models

Abstract

Large Multimodal Models (LMMs) are increasingly essential in various real-time applications, yet their substantial parameter counts and complex architectures pose significant challenges. Traditional global compression methods often rely on trial-and-error experimentation, leading to inefficiencies. In this paper, we introduce new GS-MM, an Efficient Global Sparsification technique tailored for resource-conscious multimodal models. GS-MM assigns global sparsification strategies by extracting primitive importance from the model’s components. We first derive the importance of elements from the mapping values of fully weighted activations, based on the weights of the elements. Subsequently, we compute the average multimodal importance to establish a global importance score. This score is then linearly mapped to determine the global allocation ratio, enabling the realization of global sparsity in LMMs. We demonstrate the effectiveness of our approach through extensive experiments on diverse benchmarks, including visual question-answering and reasoning tasks. Our pruned models consistently outperform conventional pruning methods, setting new standards for compressed model performance. Notably, our approach exhibits remarkable resilience to increasing sparsity ratios, preserving model quality even under extreme compression.