SalienTR: A closer look at multi-modal transformer for RGB-T salient object detection

Salient object detection (SOD) has been a fundamental task in computer vision, which aims at capturing and locating the most visually attractive object from input images. SOD is able to support and benefit various computer vision tasks, such as image segmentation (Liu et al., 2023), object detection (Jia, Song, Cao, & Lu, 2023), visual tracking (He & Chen, 2023), etc. RGB-based SOD research has achieved much progress these years (Kousik, Natarajan, Raja, Kallam, Patan, Gandomi, 2021, Peng, Yang, Li, 2022, Qi, Guo, Li, Niu, Qu, 2024, Xia, Sun, Li, Ge, Zhang, Jiang, Zhang, 2024, Yao, Wang, 2023). However, it might encounter some difficulties on condition that the background is cluttered or the light is dim, in which there could be some disturbance of texture and color details in RGB images. Fortunately, other modalities can be used to nicely complement RGB images for better performing SOD task, and the research on SOD has developed from images to multi-modal pairs (Kanwal, Taj, 2023, Wang, Wang, Sun, 2023, Wu, Sun, Xu, Meng, Wang, 2022). For instance, depth map includes more spatial and geometric information as well as three dimensional layout, which can highlight the outline of objects and facilitate the detection; thermal map reflects the thermal infrared radiation above absolute zero, which is more robust to poor illumination conditions. Additionally, severe weather does not have much impact on thermal images; consequently, thermal map can offer all-day stable SOD. As the fusion of RGB image and depth/thermal map provides a more comprehensive understanding in challenging scenarios, how to effectively design cross-modal fusion has gained continuous attention.

In terms of fusion strategies, current methods can be classified into 3 categories, which are early fusion, late fusion and multi-scale fusion. The first two types mainly focus on whether to combine the multi-modal information from the very beginning or at the end of the network, while neglecting multi-level feature interaction. Multi-scale fusion is just such a strategy that utilizes low-level, intermediate and high-level cross-modal features comprehensively, which we adopt to construct our network. The application of convolutional neural networks (CNNs) has promoted saliency detection to a large degree (Ji, Zhang, Zhang, Liu, 2021, Qu, He, Zhang, Tian, Tang, Yang, 2017), but it is limited in capturing long-range dependencies. The transformer-based architecture (Liu, Zhang, Wan, Shao, Han, 2021a, Liu, Wang, Tu, Xiao, Tang, 2021c) that we utilize naturally models global semantics, showing the capability of further pushing forward salient object detection.

Similarly, the design of transformer-based cross-modal interaction can be categorized into 4 classes, as shown in Fig. 1. Standard self-attention builds connections between modalities in an implicit way. As this relationship is relatively vague, the cross-modal complementarity may not be ideal. To make up for this deficiency, cross-attention takes the relations between each element in one modality and all elements in the other modality into account. However, this design overlooks intra-modal relations. The dense connection in fully joint attention remedies this situation. Even so, the redundant links may cause either resource consumption or potential side effects. To overcome these drawbacks, we devise a novel sparse attention mechanism to perform cross-modal interaction more effectively. It first computes self-attention in local blocks along modal-space dimensions and then adopts sparse axial global attention to build relationships among these local blocks. This strategy helps model learn complementary synergies from well-aligned RGB-T pairs, as well as boost computation efficiency.

In this paper, we propose SalienTR, a brand new encoder-decoder framework for RGB-T salient object detection. Concretely, the input image is first split into non-overlapping patches before feeding into encoder, which is comprised of two parallel Swin Transformers as the backbone network to obtain hierarchical features. A cross-modal fusion transformer, aliased as ComFormer, is designed to fuse intermediate feature maps from RGB images and the corresponding thermal maps by extending the self-attention mechanism from the image space to the space-modal 3D volume. To be more specific, a local cross-modal multi-head self-attention (LoC-MSA) module first performs sparse self-attention computation in local blocks over the space-modal volume. Then, a global cross-modal multi-head self-attention (GLoC-MSA) module adopts similar self-attention on feature patches across vertical and horizontal dimensions. Equipped with the above two modules, ComFormer is capable of capturing local cross-modal features and learning global long-range representations. Next, a uni-modal convolution (Uni-Conv) module employs convolution on RGB images and thermal maps, separately, to understand fused information as well as inject convolutional inductive bias. In order to generate high-quality segmentation masks and sharper contours, we present a dual-stream decoder to concatenate multi-modal features from ComFormer and multi-level encoded features from the backbone, and then predict salient map and the corresponding edge map. Comprehensive experiments have indicated the effectiveness and generalization of our SalienTR, which outperforms other state-of-the-art approaches. In summary, our main contributions are threefold:

• $•$
  A novel transformer-based framework, termed SalienTR, is proposed for RGB-T salient object detection task, which can effectively extract and fuse multi-level features from different modalities, and generate high-quality salient object maps and sharper object boundaries.
• $•$
  A scalable cross-modal fusion transformer, namely ComFormer, is designed to extend the self-attention mechanism from the image space to the space-modal 3D volume. It computes local attention by considering the neighboring patches across different modalities and then calculates global attention over sparse RGB-T patches along vertical and horizontal dimensions. This allows our SalienTR to better exploit the complementarity of inter- or intra-modality and capture local correlations and long-range dependencies between different modalities.
• $•$
  We conduct comprehensive experiments on three public RGB-T datasets, achieving state-of-the-art results. In particular, SalienTR exhibits higher robustness than other approaches on degraded images under challenging scenarios. Besides, SalienTR can also be applied to RGB-D SOD tasks and its generalization is further verified.