HFSENet: Hierarchical Fusion Semantic Enhancement Network for RGB-T Semantic Segmentation in Annealing Furnace Operation Area

Abstract

Regular temperature measurement of critical parts of an annealing furnace has always been a difficult task. Due to the harsh environment of high temperature, high noise, and darkness in the annealing furnace operation area, unmanned vehicles equipped with the RGB-T semantic segmentation model are usually adopted in most factories for inspection. However, existing RGB-T semantic segmentation models usually rely on good lighting or thermal conditions, which are generally difficult to fulfill in annealing furnace operation areas. In this paper, we propose a new hierarchical fusion-based semantic enhancement network, HFSENet. We first adopt the two-stream structure and the siamese structure to extract the low-level and high-level features of unimodal modalities, respectively. Then, considering the differences between the features in different hierarchical levels, we introduce a novel low-level feature spatial fusion module and a high-level feature channel fusion module to perform the multi-modal feature hierarchical fusion. On this basis, we also propose the semantic feature complementary enhancement module, which utilizes the appearance information set and object information set extracted from RGB and thermal infrared (TIR) branches to enhance the fused features and give them more semantic information. Finally, segmentation results with refined edges are obtained by an edge refinement decoder that includes a local search extraction module. The unmanned inspection vehicle we built with the proposed HFSENet has successfully passed the test, and the recognition performance of the four targets exceeds the current state-of-the-art (SOTA) method on our homemade annealing furnace operation area dataset.