Towards Maximizing Semantic Coverage for Image-Text Retrieval

Abstract

To establish semantic associations between images and texts, existing Image-Text Retrieval (ITR) methods primarily focus on fixed-scale fragments, which only identify explicit semantic categories. Consequently, semantic coverage is constrained, leading to the omission of certain semantic associations. To enlarge the semantic coverage, we propose the Semantic Coverage-Aware Network (SCA-Net). First, explicit semantic categories are identified by SCA-Net through analyzing the semantic membership of visual and textual fragments, thereby establishing more precise explicit semantic associations. Second, implicit semantic categories are identified by SCA-Net via adaptively aggregating visual and textual fragments across various scales using a co-occurrence-aware router, thereby significantly expanding the semantic coverage and establishing complete semantic associations. Third, image-text similarity is calculated using the attention mechanism over a broader range of semantic coverage. Extensive experiments demonstrate that SCA-Net significantly enhances ITR performance compared to state-of-the-art methods by maximizing semantic coverage.