Skip to content

README.md

Latest commit

 

History

History
346 lines (244 loc) · 13.2 KB

README.md

File metadata and controls

346 lines (244 loc) · 13.2 KB

🚁 FlyAwareV2: Multi-Modal UAV Dataset for Urban Semantic Segmentation

License: GPL3 Python 3.8+ Paper Dataset

The development of computer vision algorithms for Unmanned Aerial Vehicle (UAV) applications in urban environments heavily relies on the availability of large-scale datasets with accurate annotations. However, collecting and annotating real-world UAV data is extremely challenging and costly. To address this limitation, we present FlyAwareV2, a novel multimodal dataset encompassing both real and synthetic UAV imagery tailored for urban scene understanding tasks. Building upon the recently introduced SynDrone and FlyAware datasets, FlyAwareV2 introduces several new key contributions: 1) Multimodal data (RGB, depth, semantic labels) across diverse environmental conditions including varying weather and daytime; 2) Depth maps for real samples computed via state-of-the-art monocular depth estimation; 3) Benchmarks for RGB and multimodal semantic segmentation on standard architectures; 4) Studies on synthetic-to-real domain adaptation to assess the generalization capabilities of models trained on the synthetic data. With its rich set of annotations and environmental diversity, FlyAwareV2 provides a valuable resource for research on UAV-based 3D urban scene understanding.

👥 Authors

Francesco Barbato* Matteo Caligiuri* Pietro Zanuttigh

Department of Information Engineering, University of Padova, Via Gradenigo 6/b, 35131 Padova, Italy

* These authors contributed equally to this work.

📊 Graphical Abstract

FlyAwareV2 Graphical Abstract

🌟 Key Features

  • 🏙️ Multi-Environment: Multiple urban towns and scenarios
  • 📏 Multi-Altitude: Different recording heights (20m, 50m, 120m)
  • 🎯 Multi-Modal: RGB, Depth, and Semantic annotations
  • 🌦️ Adverse Weather: Sunny, Rainy, Foggy, and Night conditions
  • 🔄 Synthetic + Real: CARLA-generated synthetic data + augmented real imagery
  • 📊 Comprehensive Benchmarks: Complete evaluation suite with domain adaptation

Important

This dataset is specifically designed for adverse weather analysis in urban UAV scenarios, making it unique for studying weather-robust semantic segmentation algorithms.

🎯 Citation

If you use FlyAwareV2 in your research, please cite our paper:

@article{BARBATO2026117483,
      title = {FlyAwareV2: A multimodal cross-domain UAV dataset for urban scene understanding},
      journal = {Signal Processing: Image Communication},
      pages = {117483},
      year = {2026},
      issn = {0923-5965},
      doi = {https://doi.org/10.1016/j.image.2026.117483},
      url = {https://www.sciencedirect.com/science/article/pii/S0923596526000068},
      author = {Francesco Barbato and Matteo Caligiuri and Pietro Zanuttigh},
      keywords = {Semantic segmentation, Aerial imaging, Depth data, Multimodal scene understanding, Unsupervised domain adaptation}
}

⬇️ Dataset Download

Important

The FlyAwareV2 dataset is now available for download! Choose the version that best fits your research needs.

🔗 Download Links

Dataset Version Size Description Download
🎮 Synthetic Only ~290 GB CARLA-generated data (script to download multiple zip files) Download Synthetic
📷 Real Only ~6 GB Augmented real UAV imagery from UAVid & VisDrone (direct download) Download Real
🔄 Complete Dataset ~296 GB Both synthetic and real data (script) Download Complete

Note

Check the official download page for in-depth downlaod instruction.

📁 Recommended Folder Structure

After downloading and extracting the dataset, organize your data following this structure:

FlyAwareV2/
├── 📁 real/
│   ├── 📁 train/
│   │   ├── 📁 day/               # Clear weather training data
│   │   │   ├── 📁 rgb/           # RGB images
│   │   │   └── 📁 depth/         # Depth maps
│   │   ├── 📁 fog/               # Foggy training data
│   │   ├── 📁 night/             # Night training data
│   │   └── 📁 rain/              # Rainy training data
│   └── 📁 test/
│       ├── 📁 day/               # Test data with annotations
│       │   ├── 📁 rgb/
│       │   ├── 📁 depth/
│       │   ├── 📁 semantic/      # Semantic segmentation
│       ├── 📁 fog/
│       ├── 📁 night/
│       └── 📁 rain/
└── 📁 synthetic/
    ├── 📁 Town01_Opt_120/        # Urban environment 1
    │   ├── 📁 ClearNoon/         # Sunny conditions
    │   │   ├── 📁 height20m/     # 20m altitude
    │   │   │   ├── 📁 rgb/
    │   │   │   ├── 📁 depth/
    │   │   │   ├── 📁 semantic/
    │   │   │   └── 📁 camera/    # Camera parameters
    │   │   ├── 📁 height50m/     # 50m altitude
    │   │   └── 📁 height80m/     # 80m altitude
    │   ├── 📁 HardRainNoon/      # Rainy conditions
    │   ├── 📁 MidFoggyNoon/      # Foggy conditions
    │   └── 📁 ClearNight/        # Night conditions
    ├── 📁 Town02_Opt_120/        # Additional towns...
    ├── 📁 Town03_Opt_120/
    ├── 📁 Town04_Opt_120/
    ├── 📁 Town05_Opt_120/
    ├── 📁 Town06_Opt_120/
    ├── 📁 Town07_Opt_120/
    └── 📁 Town10HD_Opt_120/

Note

The complete folder structure contains over 100K+ images across all modalities and conditions. Each town includes 4 weather conditions and 3 altitude levels with RGB, depth, and semantic data.

📊 Dataset Statistics

Modality Weather Conditions Towns Altitudes Total Samples
RGB + Depth + Semantic Sunny, Rainy, Foggy, Night 8 Towns 3 Heights 100K+

🌤️ Weather Conditions

Weather Description Real Data Synthetic Data
☀️ Sunny Clear weather conditions Native Simulated
🌧️ Rainy Rain effects and wet surfaces Augmented Simulated
🌫️ Foggy Fog simulation with depth-aware effects Augmented Simulated
🌙 Night Low-light and artificial lighting Partially augmented Simulated

🗂️ Repository Structure

This repository contains all the code and tools for dataset generation, processing, and evaluation:

FlyAwareV2/
├── 📁 synthetic_data_generation/  # CARLA-based synthetic data generation
├── 📁 real_data_processing/       # Real data augmentation and processing
│   ├── 📁 fog/                    # Fog simulation tools
│   └── 📁 rain_and_night/         # Rain and night augmentation
├── 📁 benchmarks/                 # Comprehensive evaluation suite
├── 📁 extras/                     # Additional resources and assets
└── 📄 README.md                   # This file

🛠️ Components Overview

Component Purpose Key Technologies
Synthetic Generation Generate realistic UAV imagery Modified CARLA Simulator
Real Data Processing Augment real imagery with weather effects MonoFog, img2img-turbo
Benchmarks Model evaluation and comparison PyTorch, Domain Adaptation

🚀 Getting Started

📋 Prerequisites

  • Python 3.8+
  • PyTorch 1.9+
  • CUDA-compatible GPU (recommended)

1️⃣ Clone the Repository

git clone --recursive https://github.com/LTTM/FlyAwareV2.git
cd FlyAwareV2

2️⃣ Dataset Download

Download the dataset as described above

🔧 Usage

🎮 Synthetic Data Generation

Generate synthetic UAV data using our modified CARLA simulator:

cd synthetic_data_generation
# Follow detailed instructions in synthetic_data_generation/README.md
python run_simulation.py --config configs/urban_config.yaml

Key Features:

  • 🏙️ Multiple urban environments (8 towns)
  • 🌦️ All weather conditions simulation
  • 📐 Configurable flight altitudes
  • 🎯 Automatic semantic annotation

🌊 Real Data Augmentation

Transform clear real images into adverse weather conditions:

🌫️ Fog Generation

cd real_data_processing/fog
python clear2fog.py --input <path_to_images> --output <output_path>

🌧️ Rain & Night Augmentation

cd real_data_processing/rain_and_night
python gradio_app.py  # Interactive interface

📈 Benchmarking & Evaluation

Comprehensive model evaluation with our benchmark suite:

cd benchmarks
# Pre-training on synthetic data
python synthetic_pretrain.py --root_path <dataset_path> --config <config_file>

# Evaluation on real data
python evaluate.py --root_path <dataset_path> --model_path <checkpoint_path>

# Domain adaptation
python UDA_finetune.py --source synthetic --target real

Tip

Check the individual README files in each directory for detailed usage instructions and configuration options.

🎯 Applications

FlyAwareV2 is designed for various computer vision tasks:

  • 🔍 Semantic Segmentation: Urban scene understanding from aerial perspectives
  • 🌦️ Adverse Weather Analysis: Robust perception in challenging conditions
  • 🔄 Domain Adaptation: Bridging synthetic-to-real domain gaps
  • 🚁 UAV Navigation: Autonomous drone navigation in urban environments
  • 📊 Benchmark Studies: Standardized evaluation of aerial perception models

📚 Dataset Origins & Augmentation

🎮 Synthetic Data

  • Source: Modified CARLA Simulator
  • Enhancement: Custom urban scenarios and weather simulation
  • Coverage: 8 different towns with varied architectural styles

📷 Real Data

  • Base Datasets:
    • UAVid - Urban aerial imagery
    • VisDrone - Drone-based object detection dataset
  • Augmentation: Custom weather transformation pipeline
  • Consistency: Domain-aware augmentation preserving semantic coherence

🏆 Benchmarks & Results

Our benchmark suite evaluates models across multiple dimensions:

  • 🎯 Semantic Segmentation Performance: mIoU, accuracy metrics
  • 🌦️ Weather Robustness: Performance degradation analysis
  • 🔄 Domain Adaptation: Synthetic-to-real transfer learning
  • ⚡ Computational Efficiency: FLOPs and inference time analysis

Note

Detailed benchmark results and leaderboards are available in the official paper.

🤝 Contributing

We welcome contributions to improve FlyAwareV2! Please see our contributing guidelines:

  1. 🍴 Fork the repository
  2. 🌿 Create a feature branch
  3. 💻 Make your changes
  4. 🧪 Add tests if applicable
  5. 📝 Update documentation
  6. 🚀 Submit a pull request

📄 License

This project is licensed under the GPL-3.0 License - see the LICENSE file for details.

🙏 Acknowledgments

  • CARLA Simulator for the base simulation environment;
  • UAVid Dataset and VisDrone Dataset for real aerial imagery;
  • imag2img-turbo and FoHIS for image translation tasks;
  • marigold for depth estimation;
  • This work was partially supported by the European Union under the Italian National Recovery and Resilience Plan (NRRP) of NextGenerationEU, partnership on "Telecommunications of the Future" (PE00000001- program "RESTART").

📞 Support

For questions and support:

🚁 Advancing UAV Perception in Urban Environments 🏙️

Built with ❤️ by the MEDIALab Research Group