When CLIP Meets PHOC: A Dual-Branch Network for Historical Document Image Retrieval

Abstract

In this paper, we leverage Contrastive Language-Image Pre-training (CLIP) for Historical Document Image Retrieval (HDIR). We are largely inspired by recent advances on CLIP and its exceptional generalization capabilities, but for the first time, we tailor it to benefit HDIR. We put forward a dual-branch joint learning network based on CLIP and Pyramid Histograms of Characters (PHOC), termed CPNet, which supports out-of-vocabulary word spotting for both query-by-example and query-by-string. To achieve this, we introduce two key components: (i) a rule-based PHOC to enhance text representations by capturing detailed character-level information, and (ii) a visual prompt that allows the CLIP image encoder to adapt and extend its knowledge to our task through additional learnable prompts. Extensive experiments on benchmark datasets, including Kanjur and Geser, demonstrate that the proposed CPNet sets a new state-of-the-art in performance for HDIR, highlighting its potential for advancing research in other text-related retrieval tasks.