HiE-VL: A Large Vision-Language Model with Hierarchical Adapter for Handwritten Mathematical Expression Recognition

Abstract

Large Vision-Language Models (LVLMs) have shown impressive capabilities across various domains, but existing LVLMs have limited performance in dense perception and structured learning problems, such as Handwritten Mathematical Expression Recognition (HMER). The primary challenges stem from the complexity of formula images comprising multiple symbols and complicated inter-symbol relationships. This poses difficulties to LVLMs with locality insensitive visual encoders and structure-agnostic vision-language projectors. To overcome these challenges, we propose HiE-VL, the first LVLM for HMER containing: (1) a primitive-aware high-resolution visual encoder, (2) a hierarchical adapter, (3) a math-context enhanced large language model (LLM). Specifically, the adopted visual encoder allows locating and recognizing symbols in complex formula images. The hierarchical adapter functions as a vision-language projector to progressively capture primitive and structure information for facilitating expression decoding. The whole model is optimized in a two-stage training pipeline. In experiments on two benchmark datasets of HMER, our model achieves significantly higher performance than existing LVLMs like GPT-4V and state-of-the-art HMER models. Our codes are available at https://github.com/guohy17/HiE-VL.