SAM Adaptation with Refocused Attention and Diverse Prompts for Medical Image Segmentation

Abstract

The adaptation research of SAM in the field of medical image mainly adopts two types of methods: parameter fine-tuning and prompt engineering, but these methods face two main issues: (1)Parameter fine-tuning methods have limitations in focusing the model encoder’s attention on the foreground of medical image; (2)Prompt engineering methods have not fully utilized SAM’s ability to handle diverse types of prompts. In response to these issues, we propose the SAM-RD model, which includes a Refocused Attention module (RA), a Sparse Prompt generator (SP), and a Dense Prompt generator (DP). RA refocuses the attention within the SAM encoder to optimize its ability to capture segmentation object features. The SP and DP automatically generate diverse prompts, including positive/negative point prompts, box prompts, and mask prompts. Experiments on multiple medical datasets of different modalities show that SAM-RD outperforms the current state-of-the-art model based on SAM in medical image segmentation.