[IEEE TMM 2024] EISNet: A Multi-Modal Fusion Network for Semantic Segmentation with Events and Images

This repository is an official implementation of EISNet.

If you use any of this code, please cite the following publication:

@article{xie2024eisnet,
  title={{EISNet}: A Multi-Modal Fusion Network for Semantic Segmentation with Events and Images},
  author={Xie, Bochen and Deng, Yongjian and Shao, Zhanpeng and Li, Youfu},
  journal={IEEE Trans. Multimedia},
  volume={26},
  pages={8639--8650},
  year={2024}
}

In this project, we propose a multi-modal fusion network (EISNet) for semantic segmentation with events and images, which is comprised of two key components: Activity-Aware Event Integration Module (AEIM) and Modality Recalibration and Fusion Module (MRFM). AEIM integrates the visual cues of event data into frame-based representations that encode rich and high-confidence information via scene activity modeling. MRFM fully considers the characteristics of two modalities to achieve adaptive feature aggregation through modality recalibration and gated cross-attention fusion. Extensive experiments and ablation studies demonstrate the effectiveness and robustness of EISNet in challenging scenarios.

You can download the model weights of EISNet on the DDD17 and DSEC-Semantic datasets as follows.