This is the official PyTorch implementation for JAG 2025 paper:A novel projection map driven multimodal fusion framework for ALS point cloud segmentation
A novel projection map driven multimodal fusion framework for ALS point cloud segmentation
Abstract: Semantic segmentation of urban point clouds captured by Airborne Laser Scanning (ALS) is essential for understanding complex 3D environments, serving as a robust underlying data foundation for digital twin applications. The fusion of multimodal data has been proven to significantly improve the performance of ALS semantic segmentation by fully mining rich complementary information in each modality. However, existing fusion-based ALS semantic segmentation methods face critical limitations due to the reliance on multiple sensors, which constrains their applicability. To this end, we propose a novel multimodal framework Elevation Guidance Adaptive Fused Network, termed EGAFNet, that integrates naturally formed top-view projection images from ALS to enhance the information perception of the point cloud. Specifically, to generate highly discriminative input representation, we propose a novel projection method that accurately preserves the relative height relationships between objects and develop a Height Adaptive Scaling Module (HASM) to adaptively adjust object heights, enhancing the expressive capability of elevation information in the projection images.As for feature representation, we design a dual-branch network that effectively captures local and global context from the projection images within a large receptive field. Meanwhile, we propose an Elevation Guidance Adaptive Fusion Module (EGAFM) that adaptively fuses 2D and 3D features based on occlusion relationships to reduce feature confusion caused by occlusion in elevation projection, ensuring meaningful fusion between multimodal features. Extensive experiments on three public datasets demonstrate that our EGAFNet outperforms current state-of-the-art methods.
- 2025-10-11: our paper is accepted for publication in the International Journal of Applied Earth Observation and Geoinformation(JAG)! 🎉
The code has been trained on:
- Ubuntu 20.04 and above.
- CUDA 11.3 and above.
- Python 3.8 and above.
conda create -n pointcept python=3.8 -y
conda activate pointcept
conda install ninja -y
# Choose version you want here: https://pytorch.org/get-started/previous-versions/
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch -y
conda install h5py pyyaml -c anaconda -y
conda install sharedarray tensorboard tensorboardx yapf addict einops scipy plyfile termcolor timm -c conda-forge -y
conda install pytorch-cluster pytorch-scatter pytorch-sparse -c pyg -y
pip install torch-geometric
# spconv (SparseUNet)
# refer https://github.com/traveller59/spconv
pip install spconv-cu113
# Open3D (visualization, optional)
pip install open3d
conda install google-sparsehash -c bioconda
export C_INCLUDE_PATH=${CONDA_PREFIX}/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=${CONDA_PREFIX}/include:CPLUS_INCLUDE_PATH
pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git
We used WHU-Urban, DALES and STPLS3D for training and three datasets for evaluation.
To train the network, prepare the dataset and put it in './data/.'. Then, you use the following command:
export PYTHONPATH=./
python tools/train.py --config-file ${CONFIG_PATH} --num-gpus ${NUM_GPU} --options save_path=${SAVE_PATH}For example:
python tools/train.py --config-file configs/whu_als/semseg_spvcnn_fusion.py --num-gpus 2 --options save_path=log/spvcnn_fusion
To evaluate the network, you can use the following commands, and do not forget to modify the corresponding datapath in the config file:
export PYTHONPATH=./
python tools/test.py --config-file ${CONFIG_PATH} --num-gpus ${NUM_GPU} --options save_path=${SAVE_PATH} weight=${CHECKPOINT_PATH}
We sincerely thank the excellent projects:
