A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale

🌟 ECCV 2026 🌟

        

Markus Gross^1,2,3,📧,  Sai B. Matha¹,  Rui Song^4,5,  Viswanathan Muthuveerappan¹,  Conrad Christoph¹,  Julius Huber¹,  Daniel Cremers ^2,3

¹ Fraunhofer IVI     ² TU Munich     ³ MCML     ⁴ UCLA     ⁵ Uni Cambridge    

• [2026/07]: Firefly segmentation model released on HuggingFace 🤗
• [2026/07]: SegFly RGB-Thermal dataset released on HuggingFace 🤗
• [2026/06]: SegFly accepted to ECCV 2026 🥳
• [2026/03]: Project page online
• [2026/03]: Preprint available on arXiv

1. Abstract
2. Download SegFly Dataset
3. SegFly Dataset Documentation
4. Firefly Model: RGB and Thermal Semantic Segmentation
5. RGB-Thermal Image Registration
6. Citation
7. License

Semantic segmentation for uncrewed aerial vehicles (UAVs) is fundamental for aerial scene understanding, yet existing RGB and RGB-T datasets remain limited in scale, diversity, and annotation efficiency due to the high cost of manual labeling and the difficulties of accurate RGB-T alignment on off-the-shelf UAVs. To address these challenges, we propose a scalable geometry-driven 2D-3D-2D paradigm that leverages multi-view redundancy in high-overlap aerial imagery to automatically propagate labels from a small subset of manually annotated RGB images to both RGB and thermal modalities within a unified framework. By lifting less than 3% of RGB images into a semantic 3D point cloud and rendering it into all views, our approach enables dense pseudo ground-truth generation across large image collections, automatically producing 97% of RGB labels and 100% of thermal labels while achieving 91% and 88% annotation accuracy without any 2D manual refinement. We further extend this 2D-3D-2D paradigm to cross-modal image registration, using 3D geometry as an intermediate alignment space to obtain fully automatic, strong pixel-level RGB-T alignment with 87% registration accuracy and no hardware-level synchronization. Applying our framework to existing geo-referenced aerial imagery, we construct SegFly, a large-scale benchmark with over 20,000 high-resolution RGB images and more than 15,000 geometrically aligned RGB-T pairs spanning diverse urban, industrial, and rural environments across multiple altitudes and seasons. On SegFly, we establish the Firefly baseline for RGB and thermal semantic segmentation and show that both conventional architectures and vision foundation models benefit substantially from SegFly supervision, highlighting the potential of geometry-driven 2D-3D-2D pipelines for scalable multi-modal aerial scene understanding.

The dataset is available on HuggingFace: SegFly Dataset. Note that when you download the dataset, there is only a single split/container having all scenes combined. See 3. SegFly Dataset Documentation for more details on the actual train/val/test splits.

You can use the HuggingFace datasets library to download the dataset as follows:

1. Install dependencies:

   pip install datasets huggingface_hub

2. Download the dataset:

   from datasets import load_dataset
   
   segfly_ds = load_dataset("markus-42/SegFly")

When loaded in Python via load_dataset(), the returned object is a HuggingFace DatasetDict containing splits. Each sample contains the following features:

When downloading the dataset repository directly (e.g., via git clone or huggingface-cli download), the folder structure is organized hierarchically by modality, scene identifier, and flight altitude:

SegFly/
├── RGB/                            # RGB-only modality data (Scenes 1-9)
│   ├── scene_01/                   # Individual scene folder
│   │   ├── 30m/                    # Flight altitude folder (30 meters)
│   │   │   ├── images/             # Source RGB images (.JPG format, high resolution)
│   │   │   ├── labels/             # 8-bit semantic label masks (.png format)
│   │   │   └── metadata.csv        # Frame-specific metadata mapping (relative paths, labels, scene, altitude)
│   │   ├── 40m/                    # Flight altitude folder (40 meters)
│   │   └── 50m/                    # Flight altitude folder (50 meters)
│   ├── scene_02/
│   ...
│   └── scene_09/
└── thermal/                        # RGB-Thermal paired modality data (Scenes 3, 4, 5, 9)
    ├── scene_03/                   # Individual scene folder
    │   ├── 30m/                    # Flight altitude folder (30 meters)
    │   │   ├── images/             # Source thermal images (LWIR, .JPG format, 640x512)
    │   │   ├── RGB_aligned/        # Source RGB images geometrically registered to thermal perspective (.jpg format)
    │   │   ├── labels/             # 8-bit semantic label masks matching the thermal viewpoint (.png format)
    │   │   └── metadata.csv        # Frame-specific metadata mapping (includes paths for aligned RGB)
    │   ├── 40m/                    # Flight altitude folder (40 meters)
    │   └── 50m/                    # Flight altitude folder (50 meters)
    ├── scene_04/
    ├── scene_05/
    └── scene_09/

• images/: Contains raw, original sensor images (high-resolution RGB for the RGB modality, and 640x512 Long-Wave Infrared for the thermal modality).
• RGB_aligned/: (Thermal Modality Only) Contains RGB images that have been projected and registered to match the thermal camera's viewpoint, resolution, intrinsics, and lense distortion.
• labels/: Contains 8-bit, single-channel PNG images where each pixel's value corresponds directly to a semantic class ID (0–36).
• metadata.csv: Contains a Hugging Face ImageFolder-compatible manifest file mapping file_name (sensor image), label (segmentation mask), and RGB_aligned paths along with categorical columns for scene, altitude, and modality.

To handle rare or dynamic categories that cannot be reconstructed reliably via static 3D SfM (e.g., pedestrians, moving vehicles, cables), SegFly applies a post-processing step to map the original 22 annotations of the underlying OccuFly dataset into 15 benchmark classes plus an ignored Unlabeled category (ID 0).

• Merged Classes: Rock (ID 5) and Cable Tower (ID 22) are merged into Ground Obstacle (ID 9).
• Ignored/Unlabeled: Person (11), Bicycle (12), Cable (21), Crane (35), and boundary pixels are mapped to Unlabeled (ID 0).

We provide our aerial RGB and thermal semantic segmentation model Firefly on HuggingFace. Check the model card for detailed documentation.

A major challenge in multimodal drone perception is the alignment of RGB and thermal cameras without specialized hardware synchronization or static sensor configurations. Since these capabilities are generally available only on custom sensor rigs, existing approaches often cannot be deployed on off-the-shelf drones. SegFly resolves this entirely in software through geometry-guided registration:

1. Independent 3D Reconstructions: SFM and MVS are run independently on both the RGB and thermal sequences.
2. ICP Point Cloud Alignment: The thermal 3D point cloud is aligned to the metric RGB point cloud using the Iterative Closest Point (ICP) algorithm, yielding a rigid transform.
3. Lens-Distortion Transfer: The aligned camera poses and calibrated distortion models are used to warp the RGB images directly into the thermal camera's viewpoint, producing the registered pixel-level aligned RGB images.

This achieves a strong cross-modal semantic registration accuracy of 87.05% entirely in software, without specialized hardware sync or static sensor configurations, enabling easy multi-modal dataset scaling using off-the-shelf drone hardware.

For detailed explanations, please refer to the our SegFly paper.

If this repository or our work was helpful to you, we would appreciate citing our papers and giving the repository a star ⭐

@inproceedings{gross2026segfly,
    title={{SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale}}, 
    author={Markus Gross and Sai Bharadhwaj Matha and Rui Song and Viswanathan Muthuveerappan and Conrad Christoph and Julius Huber and Daniel Cremers},
    booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
    year={2026},
}

Since SegFly is based on the OccuFly dataset, consider citing this work as well:

@inproceedings{gross2026occufly,
    title={{OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective}}, 
    author={Markus Gross and Sai B. Matha and Aya Fahmy and Rui Song and Daniel Cremers and Henri Meess},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2026},
}

This work is licensed under the CC BY-NC-SA 4.0 license. See the LICENSE file for the full legal terms.