Advancing applications of Self-Supervised Learning in Remote Sensing data analysis (SSL-RS)

Background: The success of many machine learning applications depends on knowledge about the relationship between the input data and the task of interest (output), hindering the application of machine learning to novel tasks. End-to-end deep learning, which does not require intermediate feature engineering, has been recommended to overcome this challenge but end-to-end deep learning models require large labelled training data sets often unavailable in many medical applications. In this study, we trained machine learning models to predict paediatric hospitalization given raw photoplethysmography (PPG) signals obtained from a pulse oximeter. We trained self-supervised learning (SSL) for automatic feature extraction from PPG signals and assessed the utility of SSL in initializing end-to-end deep learning models trained on a small labelled data set with the aim of predicting paediatric hospitalization.Methods: We compared logistic regression models fitted using features extracted using SSL with end-to-end deep learning models initialized either randomly or using weights from the SSL model. We also compared the performance of SSL models trained on labelled data alone (n=1,031) with SSL trained using both labelled and unlabelled signals (n=7,578). Results: The SSL model trained on both labelled and unlabelled PPG signals produced features that were more predictive of hospitalization compared to the SSL model trained on labelled PPG only (AUC of logistic regression model: 0.78 vs 0.74). The end-to-end deep learning model had an AUC of 0.80 when initialized using the SSL model trained on all PPG signals, 0.77 when initialized using SSL trained on labelled data only, and 0.73 when initialized randomly. Conclusions: This study shows that SSL can improve the classification of PPG signals by either extracting features required by logistic regression models or initializing end-to-end deep learning models. Furthermore, SSL can leverage larger unlabelled data sets to improve performance of models fitted using small labelled data sets.

Background: The success of many machine learning applications depends on knowledge about the relationship between the input data and the task of interest (output), hindering the application of machine learning to novel tasks. End-to-end deep learning, which does not require intermediate feature engineering, has been recommended to overcome this challenge but end-to-end deep learning models require large labelled training data sets often unavailable in many medical applications. In this study, we trained self-supervised learning (SSL) models for automatic feature extraction from raw photoplethysmography (PPG) obtained using a pulse oximeter, with the aim of predicting paediatric hospitalization. Methods: We compared logistic regression models fitted using features extracted using SSL with models trained using both clinical and SSL features. In addition, we compared end-to-end deep learning models initialized randomly or using weights from the SSL models. We also compared the performance of SSL models trained on labelled data alone (n=1,031) with SSL trained using both labelled and unlabelled signals (n=7,578). Results: Logistic regression models were more predictive of hospitalization when trained on features extracted using labelled PPG signals only compared to SSL models trained on both labelled and unlabelled signals (AUC 0.83 vs 0.80). However, features extracted using SSL model trained on both labelled and unlabelled PPG signals were more predictive of hospitalization when concatenated with clinical features (AUC 0.89 vs 0.87). The end-to-end deep learning model had an AUC of 0.80 when initialized using the SSL model trained on all PPG signals, 0.77 when initialized using SSL trained on labelled data only, and 0.73 when initialized randomly. Conclusions: This study shows that SSL can extract features from PPG signals that are predictive of hospitalization or initialize end-to-end deep learning models. Furthermore, SSL can leverage larger unlabelled data sets to improve performance of models fitted using small labelled data sets.

Machine Learning and Deep Learning in Remote Sensing Data Analysis

In the recent decades, remote sensing data are rapidly growing in size and variety. In the traditional remote sensing analysis workflows, Earth science data users have to download raw data files to local workstations before processing them for science discoveries. The data transfer often costs a lot of time and slows down the analysis workflows. In most cases, results of remote sensing data analysis models are usually much smaller than raw data to be processed. Therefore, “on-demand processing”, which tries to upload data analysis models and execute them “near” data, can reduce the costs of data transfer and significantly accelerate the execution of remote sensing analysis workflows. In this paper, a framework, which enables on-demand remote sensing data analysis, is proposed. The evaluation on a prototype system shows that on-demand processing framework obviously accelerates the execution of analysis models by reducing data transfers, especially for those analysis workflows which transfer data through low bandwidth Internet.

Modelling species distributions with remote sensing data: bridging disciplinary perspectives

Dr. Thomas Hilker left us on 4 September 2016 following a sudden cardiac arrest. Thomas was adevoted husband to Yhasmin, and a brother and son to his family in Germany to whom we expressour deepest sympathies. Friends and colleagues of Thomas in the remote sensing and ecologicalcommunities lament this tragic loss. During his short but stellar science career, Thomas becamea world leader in the field of carbon, water and energy exchange from the land. He pioneeredstudies in the Amazonian forests, using anisotropy information acquired from satellites to describethree-dimensional structures that linked these ecosystems functionally to climatic variation.Thomas had an extreme range of interests—from the engineering of advanced spectrometers toproviding new theories and innovative methods to process remotely sensed data. Dr. Piers Sellers,Acting Director of the Earth Sciences Division at NASA/GSFC, and Deputy Director of the Sciences andExploration Directorate wrote: “Thomas Hilker was something of a renaissance man in Earth Science.He could climb towers, measure tall trees, and calculate spectral indices in his head. Working with himwas like collaborating with two or three normal people. He had some of the best and most originalideas in remote sensing that I’ve come across, but unlike most of us, he could go get the data to provehis point. And he was always the best fun. I remember him coming to a couple of parties of ours—hewas always relaxed, humorous, charming. He could make people laugh and everyone felt so goodaround him.”Thomas obtained a Bachelor of Science degree in Forestry from the University of Applied Sciences,in Goettingen, Germany in 2000, a Master in Photogrammetry and Geoinformatics from the Universityof Applied Sciences in Stuttgart in 2002 and a PhD from the University of British Columbia (UBC) inForestry in 2008. After a three year postdoctoral position at UBC (2008–2011), he worked as a ResearchAssociate at NASA’s Goddard Space Flight Center (2011–2012). From 2012 to 2016, Thomas held aposition as Assistant Professor at Oregon State University’s College of Forestry, leading the RemoteSensing Laboratory and teaching classes in Remote Sensing and Spatial Data Analysis. In 2015 and2016 he was a visiting researcher at the National Institute for Space Studies in Brazil (Instituto Nacionalde Pesquisas Espaciais, INPE). He looked forward to starting a position as an Associate Professor ofEarth System Science and Remote Sensing at the University of Southampton, UK.Thomas became fascinated by the global carbon cycle following receipt of his Master’s degree andwas keen to utilize his geospatial skills to unlock the details of the cycle. Inspired by the linksbetween canopy reflectance and photosynthesis, Thomas designed an Automated MultiangularSpectroradiometer for Estimation of Canopy reflectance [1] and improved it in subsequentiterations [2,3] to be able to continuously monitor subtle changes in the reflected spectra from

The monitoring of restoration and forestation is essential to reduce future drought and flood risk as well as ongoing carbon sequestration projects in Iceland. This is especially relevant for Iceland’s efforts to become carbon neutral by 2040. Such a monitoring can be done by using the state-of-art remote sensing technology, using remotely sensed data and digital mapping approaches. The LanDeg project will use free Geographic Information System (GIS) and Remote Sensing (RS) data to map soil degradation, restoration and ongoing forestation efforts to assess carbon sequestration. For this purpose, we will validate GIS and RS data analysis with field mapping of vegetation and soil cover in a restored area in southern Iceland. The validated GIS and RS analysis will be used to assess restoration efforts and trends in vegetation cover in the area. Subsequently, the changes in the vegetation cover will be used to assess the carbon sequestration rate. Based on these results we will identify best-restoration and carbon sequestration practices.

Artificial intelligence (AI) methods have emerged as a powerful tool to study and in some cases forecast natural disasters [1,2]. Recent works have successfully combined deep learning modeling with scientific knowledge stemming from the SAR Interferometry domain propelling research on tasks like volcanic activity monitoring [3], associated with ground deformation. A milestone in this interdisciplinary field has been the release of the Hephaestus [4] InSAR dataset, facilitating automatic InSAR interpretation, volcanic activity localization as well as the detection and categorization of atmospheric contributions in wrapped interferograms. Hephaestus contains annotations for approximately 20,000 InSAR frames, covering the 44 most active volcanoes in the world. The annotation was performed  by a team of InSAR experts that manually examined each InSAR frame individually. However, even with such a large dataset, class imbalance remains a challenge, i.e. the InSAR samples containing volcano deformation fringes are orders of magnitude less than those that do not. This is anticipated since natural hazards are in principle rare in nature. To counter that, the authors of Hephaestus provide more than 100,000 unlabeled InSAR frames to be used for global large-scale self-supervised learning, which is more robust to class imbalance when compared to supervised learning [5]. Motivated by the Hephaestus dataset and the insights provided by [2], we train global, task-agnostic models in a self-supervised learning fashion that can handle distribution shifts caused by spatio-temporal variability as well as major class imbalances. By finetuning such a model to the labeled part of Hephaestus we obtain the backbone for a global volcanic activity alerting system, namely Pluto. Pluto is a novel end-to-end AI based system that provides early warnings of volcanic unrest on a global scale.Pluto automatically synchronizes its database with the Comet-LiCS [6] portal to receive newly generated Sentinel-1 InSAR data acquired over volcanic areas. The new samples are fed to our volcanic activity detection model. If volcanic activity is detected, an automatic email is sent to the service users, which contains information about the intensity, the exact location and the type (Mogi, Sill, Dyk) of the event. To ensure a robust and ever-improving service we augment Pluto with an iterative pipeline that collects samples that were misclassified in production, and uses them to further improve the existing model.  [1] Kondylatos et al. "Wildfire danger prediction and understanding with Deep Learning." Geophysical Research Letters 49.17 (2022): e2022GL099368.[2] Bountos et al. "Self-supervised contrastive learning for volcanic unrest detection." IEEE Geoscience and Remote Sensing Letters 19 (2021): 1-5.[3] Bountos et al. "Learning from Synthetic InSAR with Vision Transformers: The case of volcanic unrest detection." IEEE Transactions on Geoscience and Remote Sensing (2022).[4] Bountos et al. "Hephaestus: A large scale multitask dataset towards InSAR understanding." Proceedings of the IEEE/CVF CVPR. 2022.[5] Liu et al. "Self-supervised learning is more robust to dataset imbalance." arXiv preprint arXiv:2110.05025 (2021).[6] Lazecký et al. "LiCSAR: An automatic InSAR tool for measuring and monitoring tectonic and volcanic activity." Remote Sensing 12.15 (2020): 2430.

Traditional geological mapping methods, which rely on field observations and rock sample analysis, are inefficient for continuous spatial mapping of geological features such as alteration zones. Deep learning models such as convolutional neural networks (CNNs) have ushered in a transformative era in remote sensing data analysis. CNNs excel in automatically extracting features from image data for classification and regression problems. CNNs have the ability to pinpoint specific mineralogical changes attributed to mineralisation processes by discerning subtle features within remote sensing data. In this study, we deploy CNNs with three sets of remote sensing data, namely Landsat 8, Landsat 9, and ASTER, to delineate diverse alteration zones within a mineral-rich region north of Broken Hill in western New South Wales, Australia. Our methodology involves model training using two distinct sets of training samples generated through ground truth data and a fully automated approach through selective principal component analysis (PCA). We also compare CNNs with conventional machine learning models, including k‑nearest neighbours, support vector machines, and multilayer perceptron. Our findings indicate that training with a ground truth-based dataset produces more reliable alteration maps. Additionally, we find that CNNs perform slightly better when compared to conventional machine learning models, which further demonstrates the ability of CNNs to capture spatial patterns in remote sensing data effectively. We find that Landsat 9 surpasses Landsat 8 in mapping iron oxide areas when employing the CNNs model trained with ground truth data obtained by field surveys. We also observe that using ASTER data with the CNNs model trained on the ground truth-based dataset produces the most accurate maps for two other important types of alteration zones, argillic and propylitic. This underscores the utility of CNNs in enhancing the efficiency and precision of geological mapping, particularly in discerning subtle alterations indicative of mineralisation processes, especially those associated with critical metal resources.

Remote sensing data has been widely used for various Earth Observation (EO) missions such as land use and cover classification, weather forecasting, agricultural management, and environmental monitoring. Most existing remote-sensing-data-based models are based on supervised learning that requires large and representative human-labeled data for model training, which is costly and time-consuming. The recent introduction of self-supervised learning (SSL) enables models to learn a representation from orders of magnitude more unlabeled data. The success of SSL is heavily dependent on a pre-designed pretext task, which introduces an inductive bias into the model from a large amount of unlabeled data. Since remote sensing imagery has rich spectral information beyond the standard RGB color space, it may not be straightforward to extend to the multi/hyperspectral domain the pretext tasks established in computer vision based on RGB images. To address this challenge, this work proposed a generic self-supervised learning framework based on remote sensing data at both the object and pixel levels. The method contains two novel pretext tasks, one for object-based and one for pixel-based remote sensing data analysis methods. One pretext task is used to reconstruct the spectral profile from the masked data, which can be used to extract a representation of pixel information and improve the performance of downstream tasks associated with pixel-based analysis. The second pretext task is used to identify objects from multiple views of the same object in multispectral data, which can be used to extract a representation and improve the performance of downstream tasks associated with object-based analysis. The results of two typical downstream task evaluation exercises (a multilabel land cover classification task on Sentinel-2 multispectral datasets and a ground soil parameter retrieval task on hyperspectral datasets) demonstrate that the proposed SSL method learns a target representation that covers both spatial and spectral information from massive unlabeled data. A comparison with currently available SSL methods shows that the proposed method, which emphasizes both spectral and spatial features, outperforms existing SSL methods on multi- and hyperspectral remote sensing datasets. We believe that this approach has the potential to be effective in a wider range of remote sensing applications and we will explore its utility in more remote sensing applications in the future.

Remote sensing images (RSI) are significant data to examine and observe complete structure on the Earth’s surface. RSI classification has gained significant attention in earth observation technologies, commonly employed in military and civil fields. It becomes a challenging process because of high dimensional features and small amount of labeled data. Advancements in machine learning (ML) and deep learning (DL) models are capable in effective RSI classification. Numerous research is going on in RSI detection and classification area using ML and DL models. In this view, this article focuses on the review of recently developed RSI classification models. A brief introduction to RSI, types, characteristics and challenging issues is given. By a meta-analysis, different approaches related to the RSI classification models are identified and summarized with key findings. Besides, this survey covers the recently developed ML and DL based RSI classification models with their major aim, methodology used, merits, and demerits. At last, a concluding remark related to the present state of art approaches with possible future scope is discussed.

Deep learning (DL) finds rich applications in the wireless domain to improve spectrum awareness. Typically, DL models are either randomly initialized following a statistical distribution or pretrained on tasks from other domains in the form of transfer learning without accounting for the unique characteristics of wireless signals. Self-supervised learning (SSL) enables the learning of useful representations from Radio Frequency (RF) signals themselves even when only limited training data samples with labels are available. We present a self-supervised RF signal representation learning method and apply it to the automatic modulation recognition (AMR) task by specifically formulating a set of transformations to capture the wireless signal characteristics. We show that the sample efficiency (the number of labeled samples needed to achieve a certain performance) of AMR can be significantly increased (almost an order of magnitude) by learning signal representations with SSL. This translates to substantial time and cost savings. Furthermore, SSL increases the model accuracy compared to the state-of-the-art DL methods and maintains high accuracy when limited training data is available.

Multimodal learning, which aims to understand and analyze information from multiple modalities, has achieved substantial progress in the supervised regime in recent years. However, the heavy dependence on data paired with expensive human annotations impedes scaling up models. Meanwhile, given the availability of large-scale unannotated data in the wild, self-supervised learning has become an attractive strategy to alleviate the annotation bottleneck. Building on these two directions, self-supervised multimodal learning (SSML) provides ways to learn from raw multimodal data. In this survey, we provide a comprehensive review of the state-of-the-art in SSML, in which we elucidate three major challenges intrinsic to self-supervised learning with multimodal data: 1) learning representations from multimodal data without labels, 2) fusion of different modalities, and 3) learning with unaligned data. We then detail existing solutions to these challenges. Specifically, we consider 1) objectives for learning from multimodal unlabeled data via self-supervision, 2) model architectures from the perspective of different multimodal fusion strategies, and 3) pair-free learning strategies for coarse-grained and fine-grained alignment. We also review real-world applications of SSML algorithms in diverse fields, such as healthcare, remote sensing, and machine translation. Finally, we discuss challenges and future directions for SSML.

    Morocco has become one of the most industrialized feet, not only in Africa but also in the world. This mutation benefits the country's development; however, these industrial activities directly or indirectly impact the impairment grade environment. Remote Sensing (RS) data offer consideration from government projects and commercial applications to academic fields from free-open access data centers. The EUMETSAT NASA, NOAA, ESA, Copernicus, etc., deliver RS data with numerous satellites flying on geostationary and polar orbits. The provided data are massive (Terabytes daily) for environmental monitoring, disaster management, and other applications. RS products are measured with various instruments, for instance, radiometer, spectrometer, hyperspectral, sounder, altimeter, and optical. Wide spectral bands are employed, such as infrared, visible, radar, microwave, etc. The proliferation of RS data also increases the RS data's velocity (thousands of files daily), the data's diversity (NetCDF, HDF5, BUFR, binary, etc.), and higher dimensionality characteristics. Accordingly, RS data can be regarded as Big Data (BD). Thus, it is challenging to acquire, ingest, process, store, query, and visualize RSBD proficiently due to the data and computing-intensive challenges and limitations. As a result, an incredible deal of attention in the field of BD and its analysis has increased, most ambitious from a vast number of research challenges powerfully related to RS applications, such as modeling, pre-processing, analyzing, querying, and mining, in distributed and scalable clusters.    This project aims to design and develop an African Earth Open Portal (AfEOP) as a platform for the automatic acquisition, ingestion, processing, and visualization of the massive stream of RS datasets from multiples satellites sensors, ground stations, drones, etc. The proposed platform will solve many environmental issues, notably: (1) supervising the weather parameters in Africa, including the temperature, humidity, pressure, and wind speed, etc. (2) drought assessment, evapotranspiration estimation, water drainage monitoring, and food yield and crop forecasting 3) agriculture and fertilization optimization using Artificial Intelligence (AI) algorithms helping in decision making 4) water use reduction using RS data, AI, and physical models.    In this project, we perform BD Analytics by (1) presenting a brief survey of the used data sources and describing the nature of the used satellite sensors' data for environmental applications. (2) designing and developing an ingestion framework for RSBD in a distributed platform for RS data storage and query. (3) incorporating cloud computing and parallel programming techniques to optimize processing. (4) visualizing the results in demand in interactive maps and dashboards. Accordingly, this led us to the following questions: Is the designed architecture of RS data pre-processing efficient to extract only helpful information in a short execution time? Is it possible to make the RS data-friendly with the distributed framework for more processing? Are RS techniques efficient for environmental applications for Africa and notably Morocco?

With its tremendous success in many machine learning and pattern recognition tasks, deep learning, as one type of data-driven models, has also led to many breakthroughs in other disciplines including physics, chemistry and material science. Nevertheless, the supremacy of deep learning over conventional optimization approaches heavily depends on the huge amount of data collected in advance to train the model, which is a common bottleneck of such a data-driven technique. In this work, we present a comprehensive deep learning model for the design and characterization of nanophotonic structures, where a self-supervised learning mechanism is introduced to alleviate the burden of data acquisition. Taking reflective metasurfaces as an example, we demonstrate that the self-supervised deep learning model can effectively utilize randomly generated unlabeled data during training, with the total test loss and prediction accuracy improved by about 15% compared with the fully supervised counterpart. The proposed self-supervised learning scheme provides an efficient solution for deep learning models in some physics-related tasks where labeled data are limited or expensive to collect.