SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation (2211.07044v2)

Abstract: Self-supervised pre-training bears potential to generate expressive representations without human annotation. Most pre-training in Earth observation (EO) are based on ImageNet or medium-size, labeled remote sensing (RS) datasets. We share an unlabeled RS dataset SSL4EO-S12 (Self-Supervised Learning for Earth Observation - Sentinel-1/2) to assemble a large-scale, global, multimodal, and multi-seasonal corpus of satellite imagery from the ESA Sentinel-1 & -2 satellite missions. For EO applications we demonstrate SSL4EO-S12 to succeed in self-supervised pre-training for a set of methods: MoCo-v2, DINO, MAE, and data2vec. Resulting models yield downstream performance close to, or surpassing accuracy measures of supervised learning. In addition, pre-training on SSL4EO-S12 excels compared to existing datasets. We make openly available the dataset, related source code, and pre-trained models at https://github.com/zhu-xlab/SSL4EO-S12.

• The paper introduces the SSL4EO-S12 dataset, featuring 250k global locations with four seasonal snapshots per location to advance self-supervised learning in Earth observation.
• It leverages multimodal data from Sentinel-1 and Sentinel-2, demonstrating significant improvements in tasks like scene classification and semantic segmentation.
• The study confirms that unsupervised pre-training on this diverse dataset markedly enhances downstream remote sensing applications and informs future dataset expansion.

Overview of SSL4EO-S12 Dataset for Self-Supervised Learning in Earth Observation

The SSL4EO-S12 dataset advances the domain of Earth observation (EO) by providing a robust multi-modal and multi-temporal set for self-supervised learning (SSL). Addressing the limitations of existing datasets—such as the domain gap when using ImageNet for remote sensing tasks and restricted geographical diversity in datasets like SEN12MS—this paper introduces a dataset specifically tailored to encompass the vast and varied data types present in EO.

Key Contributions

The primary contribution of the paper lies in the development of SSL4EO-S12, which includes:

• Scale and Diversity: Sampling 250k locations worldwide, it incorporates four seasonal snapshots per location, enhancing the spatial and temporal diversity. The dataset comprises Sentinel-1 and Sentinel-2 images, which encompass SAR and optical modalities, ensuring a comprehensive multimodal coverage that improves representation learning.
• Geographical Extent and Coverage: Unlike previous remote sensing datasets, SSL4EO-S12 minimizes geographical overlap and focuses on global coverage, thereby capturing a multitude of climates and environments which enrich the dataset's applicability.
• Unsupervised Pre-training: By demonstrating the efficacy of SSL on EO data using standard algorithms like MoCo-v2, DINO, MAE, and data2vec, the dataset is shown to improve downstream performance on tasks like scene classification, semantic segmentation, and change detection significantly.

Evaluation and Results

The experimental section of the paper details an extensive benchmarking effort:

• Pre-training Validation: The SSL4EO-S12 dataset enabled superior performance compared to prior datasets, underscoring the importance of a large-scale, multimodal, and multi-temporal approach. Particularly, it achieved notable results across diverse SSL frameworks, proving its versatility and robustness as a pre-training resource.
• Downstream Task Improvement: Significant enhancement in performance was observed across various downstream applications. For instance, when used for scene classification on datasets like EuroSAT, linear probing and fine-tuning approaches demonstrated marked improvement over traditional supervised methods.
• Ablation Studies: The studies highlight the dataset’s strengths, particularly the advantages of multimodal pre-training and the added value of seasonal and atmospheric data augmentation—enhancements that underscore its vital role in capturing complex data characteristics within EO.

Implications and Future Work

The SSL4EO-S12 dataset represents an important step toward optimizing SSL for EO applications. By emphasizing large-scale, multimodal datasets, the paper signals a shift from domain-specific datasets towards more generalized approaches, which could lead to significant advancements in remote sensing technologies. Future developments may include further expansion of the dataset to include other sensor types, enhancing EO systems’ adaptability and robustness to diverse aerial and terrestrial conditions. Moreover, as SSL algorithms advance, SSL4EO-S12 could play a pivotal role in fostering innovations that require less labeled data, aligning with trends in unsupervised and semi-supervised learning paradigms.

The research illuminates a pathway towards more effective and efficient EO data processing and invites further exploration into the optimization of SSL methodologies within this rich data context. It is anticipated that the principles and practices generalized from this dataset will inspire continued innovation and application in remote sensing and beyond.