Earth Observation Data Processing Research Articles

OpenEOcubes: an open-source and lightweight R-based RESTful web service for analyzing earth observation data cubes

In recent decades, Earth Observation (EO) systems have seen remarkable technological advancements, leading to a surge in Earth-orbiting satellites capturing EO data. Cloud-based storage solutions have been adopted to manage the increasing data volume. Although numerous EO data management and analysis platforms have emerged to accommodate this growth, many suffer from limitations like closed-source software, leading to platform lock-in and restricted functionalities, restricting the scientific community from conducting open and reproducible research. To tackle these issues, we present OpenEOcubes, a lightweight EO data cubes analysis service that embraces open-source tools, standardized APIs, and containerized deployment, we demonstrate the service’s capabilities in two user scenarios: performing vegetation analysis in Amazonia, Brazil for one year, and detecting changes in a forested area in Brandenburg, Germany based on five years of EO data.OpenEOcubes is an easy-to-deploy service that empowers the scientific community to reproduce small and medium-sized EO scientific analysis while aggregating over a potentially huge amount of data. It enables the extension of functionalities and validation of analysis carried out on different EO data processing platforms.

Scalable Data Processing Platform for Earth Observation Data Repositories

Earth observation (EO) satellite data is essential to environmental monitoring. At a national and regional level, the open data cubes harness the power of satellite data by providing application programming interfaces and services to the end-users. The volume and the complexity of satellite observations are increasing, demanding novel approaches for data storing, managing, and processing. High-performance computing (HPC) and cloud platforms may improve Big EO data processing performance. However, it is necessary to consider several vital aspects for efficient and flexible EO data processing, such as the interoperability from cloud-HPC and EO data repositories, automatic provisioning and scaling of cloud-HPC resources, cost-effectiveness, support of new EO data formats and open-source packages, or linkage with data cube platforms. The article proposes a scalable EO data processing platform interoperable from cloud-HPC and EO data repositories. The platform enables linking any data repository supporting web coverage service or SpatioTemporal Asset Catalog Application Programming Interfaces (STAC-API), and any cloud or HPC resource supporting scheduling system API for providing access to the cluster backends.

A Multi-Objective Optimization Service for Enhancing Performance and Cost Efficiency in Earth Observation Data Processing Workflows

A Multi-Objective Optimization Service for Enhancing Performance and Cost Efficiency in Earth Observation Data Processing Workflows

A Python Library for the Jupyteo IDE Earth Observation Processing Tool Enabling Interoperability with the QGIS System for Use in Data Science

This paper describes JupyQgis – a new Python library for Jupyteo IDE enabling interoperability with the QGIS system. Jupyteo is an online integrated development environment for earth observation data processing and is available on a cloud platform. It is targeted at remote sensing experts, scientists and users who can develop the Jupyter notebook by reusing embedded open-source tools, WPS interfaces and existing notebooks. In recent years, there has been an increasing popularity of data science methods that have become the focus of many organizations. Many scientific disciplines are facing a significant transformation due to data-driven solutions. This is especially true of geodesy, environmental sciences, and Earth sciences, where large data sets, such as Earth observation satellite data (EO data) and GIS data are used. The previous experience in using Jupyteo, both among the users of this platform and its creators, indicates the need to supplement its functionality with GIS analytical tools. This study analyzed the most efficient way to combine the functionality of the QGIS system with the functionality of the Jupyteo platform in one tool. It was found that the most suitable solution is to create a custom library providing an API for collaboration between both environments. The resulting library makes the work much easier and simplifies the source code of the created Python scripts. The functionality of the developed solution was illustrated with a test use case.

WaCoDiS: Automated Earth Observation data processing within an event-driven architecture for water monitoring

Climate changes and the ongoing intensification of agriculture effect in increased material inputs in watercourses and dams. To ensure an efficient and environmentally friendly water resource management, water management associations need means for efficient water monitoring as well as novel strategies to reduce the pollution of surface and ground water. Traditionally, water management associations operate large sensor networks to suffice their needs for hydrological and meteorological measurement data to monitor and model physical processes within the catchments they are responsible for. Implementing a comprehensive monitoring system often suffers from sparse coverage of in-situ data. Due to the evolvement of the Copernicus satellite platforms, the broader availability of satellite data provides a great potential for deriving valuable, complementary information from Earth Observation data, that contributes to a detailed understanding of hydrological processes. Although the number of satellite data platforms that provide online processing environments is growing, it is still a big challenge to integrate those platforms into traditional workflows of users from environmental domains such as hydrology. Thus, in this paper, we introduce a software architecture to facilitate the generation of Earth Observation information targeted towards hydrology. The presented WaCoDiS System comprises several microservices as well standardized interfaces that enable a platform-independent processing of satellite data. First, we discuss the contribution of Earth Observation data to water monitoring and derive several challenges regarding the facilitation of satellite data processing. We then describe our system design with a brief overview about the different system components which form an automated processing pipeline. The suitability of our system is proven as part of a pre-operational deployment for a German water management association. In addition, we demonstrate how our system is capable of integrating satellite data platforms, using the Copernicus Data and Exploitation Platform - Deutschland (CODE-DE) as a reference example.

ISO Cluster classifier by ArcGIS for unsupervised classification of the Landsat TM image of Reykjavík

The paper presents the use of the Landsat TM image processed by the ArcGIS Spatial Analyst Tool for environmental mapping of southwestern Iceland, region of Reykjavik. Iceland is one of the most special Arctic regions with unique flora and landscapes. Its environment is presented by vulnerable ecosystems of highlands where vegetation is affected by climate, human or geologic factors: overgrazing, volcanism, annual temperature change. Therefore, mapping land cover types in Iceland contribute to the nature conservation, sustainable development and environmental monitoring purposes. This paper starts by review of the current trends in remote sensing, the importance of Landsat TM imagery for environmental mapping in general and Iceland in particular, and the requirements of GIS specifically for satellite image analysis. This is followed by the extended methodological workflow supported by illustrative print screens and technical description of data processing in ArcGIS. The data used in this research include Landsat TM image which was captured using GloVis and processed in ArcGIS. The methodology includes a workflow involving several technical steps of raster data processing in ArcGIS: 1) coordinate projecting, 2) panchromatic sharpening, 3) inspection of raster statistics, 4) spectral bands combination, 5) calculations, 6) unsupervised classification, 7) mapping. The classification was done by clustering technique using ISO Cluster algorithm and Maximum Likelihood Classification. This paper finally presents the results of the ISO Cluster application for Landsat TM image processing and concludes final remarks on the perspectives of environmental mapping based on Landsat TM image processing in ArcGIS.The results of the classification present landscapes divided into eight distinct land cover classes: 1) bare soils; 2) shrubs and smaller trees in the river valleys, urban areas including green spaces; 3) water areas; 4) forests including the Reykjanesfólkvangur National reserve; 5) ice-covered areas, glaciers and cloudy regions; 6) ravine valleys with a sparse type of the vegetation: rowan, alder, heathland, wetland; 7) rocks; 8) mixed areas. The final remarks include the discussion on the development of machine learning methods and opportunities of their technical applications in GIS-based analysis and Earth Observation data processing in ArcGIS, including image analysis and classification, mapping and visualization, machine learning and environmental applications for decision making in forestry and sustainable development.

Automated Production of a Land Cover/Use Map of Europe Based on Sentinel-2 Imagery

Up-to-date information about the Earth’s surface provided by land cover maps is essential for numerous environmental and land management applications. There is, therefore, a clear need for the continuous and reliable monitoring of land cover and land cover changes. The growing availability of high resolution, regularly collected remote sensing data can support the increasing number of applications that require high spatial resolution products that are frequently updated (e.g., annually). However, large-scale operational mapping requires a highly-automated data processing workflow, which is currently lacking. To address this issue, we developed a methodology for the automated classification of multi-temporal Sentinel-2 imagery. The method uses a random forest classifier and existing land cover/use databases as the source of training samples. In order to demonstrate its operability, the method was implemented on a large part of the European continent, with CORINE Land Cover and High-Resolution Layers as training datasets. A land cover/use map for the year 2017 was produced, composed of 13 classes. An accuracy assessment, based on nearly 52,000 samples, revealed high thematic overall accuracy (86.1%) on a continental scale, and average overall accuracy of 86.5% at country level. Only low-frequency classes obtained lower accuracies and we recommend that their mapping should be improved in the future. Additional modifications to the classification legend, notably the fusion of thematically and spectrally similar vegetation classes, increased overall accuracy to 89.0%, and resulted in ten, general classes. A crucial aspect of the presented approach is that it embraces all of the most important elements of Earth observation data processing, enabling accurate and detailed (10 m spatial resolution) mapping with no manual user involvement. The presented methodology demonstrates possibility for frequent and repetitive operational production of large-scale land cover maps.

Employing a Multi-Input Deep Convolutional Neural Network to Derive Soil Clay Content from a Synergy of Multi-Temporal Optical and Radar Imagery Data

Earth observation (EO) has an immense potential as being an enabling tool for mapping spatial characteristics of the topsoil layer. Recently, deep learning based algorithms and cloud computing infrastructure have become available with a great potential to revolutionize the processing of EO data. This paper aims to present a novel EO-based soil monitoring approach leveraging open-access Copernicus Sentinel data and Google Earth Engine platform. Building on key results from existing data mining approaches to extract bare soil reflectance values the current study delivers valuable insights on the synergistic use of open access optical and radar images. The proposed framework is driven by the need to eliminate the influence of ambient factors and evaluate the efficiency of a convolutional neural network (CNN) to effectively combine the complimentary information contained in the pool of both optical and radar spectral information and those form auxiliary geographical coordinates mainly for soil. We developed and calibrated our multi-input CNN model based on soil samples (calibration = 80% and validation 20%) of the LUCAS database and then applied this approach to predict soil clay content. A promising prediction performance (R2 = 0.60, ratio of performance to the interquartile range (RPIQ) = 2.02, n = 6136) was achieved by the inclusion of both types (synthetic aperture radar (SAR) and laboratory visible near infrared–short wave infrared (VNIR-SWIR) multispectral) of observations using the CNN model, demonstrating an improvement of more than 5.5% in RMSE using the multi-year median optical composite and current state-of-the-art non linear machine learning methods such as random forest (RF; R2 = 0.55, RPIQ = 1.91, n = 6136) and artificial neural network (ANN; R2 = 0.44, RPIQ = 1.71, n = 6136). Moreover, we examined post-hoc techniques to interpret the CNN model and thus acquire an understanding of the relationships between spectral information and the soil target identified by the model. Looking to the future, the proposed approach can be adopted on the forthcoming hyperspectral orbital sensors to expand the current capabilities of the EO component by estimating more soil attributes with higher predictive performance.

Retrieval of Snow Properties from the Sentinel-3 Ocean and Land Colour Instrument

The Sentinel Application Platform (SNAP) architecture facilitates Earth Observation data processing. In this work, we present results from a new Snow Processor for SNAP. We also describe physical principles behind the developed snow property retrieval technique based on the analysis of Ocean and Land Colour Instrument (OLCI) onboard Sentinel-3A/B measurements over clean and polluted snow fields. Using OLCI spectral reflectance measurements in the range 400–1020 nm, we derived important snow properties such as spectral and broadband albedo, snow specific surface area, snow extent and grain size on a spatial grid of 300 m. The algorithm also incorporated cloud screening and atmospheric correction procedures over snow surfaces. We present validation results using ground measurements from Antarctica, the Greenland ice sheet and the French Alps. We find the spectral albedo retrieved with accuracy of better than 3% on average, making our retrievals sufficient for a variety of applications. Broadband albedo is retrieved with the average accuracy of about 5% over snow. Therefore, the uncertainties of satellite retrievals are close to experimental errors of ground measurements. The retrieved surface grain size shows good agreement with ground observations. Snow specific surface area observations are also consistent with our OLCI retrievals. We present snow albedo and grain size mapping over the inland ice sheet of Greenland for areas including dry snow, melted/melting snow and impurity rich bare ice. The algorithm can be applied to OLCI Sentinel-3 measurements providing an opportunity for creation of long-term snow property records essential for climate monitoring and data assimilation studies—especially in the Arctic region, where we face rapid environmental changes including reduction of snow/ice extent and, therefore, planetary albedo.

JupyTEP IDE as an Online Tool for Earth Observation Data Processing

The paper describes a new tool called JupyTEP integrated development environment (IDE), which is an online integrated development environment for earth observation data processing available in the cloud. This work is a result of the project entitled “JupyTEP IDE—Jupyter-based IDE as an interactive and collaborative environment for the development of notebook style EO algorithms on network of exploitation platforms infrastructure” carried out in cooperation with European Space Agency. The main goal of this project was to provide a universal earth observation data processing tool to the community. JupyTEP IDE is an extension of Jupyter software ecosystem with customization of existing components for the needs of earth observation scientists and other professional and non-professional users. The approach is based on configuration, customization, adaptation, and extension of Jupyter, Jupyter Hub, and Docker components on earth observation data cloud infrastructure in the most flexible way; integration with accessible libraries and earth observation data tools (sentinel application platform (SNAP), geospatial data abstraction library (GDAL), etc.); adaptation of existing web processing service (WPS)-oriented earth observation services. The user-oriented product is based on a web-related user interface in the form of extended and modified Jupyter user interface (frontend) with customized layout, earth observation data processing extension, and a set of predefined notebooks, widgets, and tools. The final IDE is addressed to the remote sensing experts and other users who intend to develop Jupyter notebooks with the reuse of embedded tools, common WPS interfaces, and existing notebooks. The paper describes the background of the system, its architecture, and possible use cases.

Paving the Way to Increased Interoperability of Earth Observations Data Cubes

Earth observations data cubes (EODCs) are a paradigm transforming the way users interact with large spatio-temporal Earth observation (EO) data. It enhances connections between data, applications and users facilitating management, access and use of analysis ready data (ARD). The ambition is allowing users to harness big EO data at a minimum cost and effort. This significant interest is illustrated by various implementations that exist. The novelty of the approach results in different innovative solutions and the lack of commonly agreed definition of EODC. Consequently, their interoperability has been recognized as a major challenge for the global change and Earth system science domains. The objective of this paper is preventing EODC from becoming silos of information; to present how interoperability can be enabled using widely-adopted geospatial standards; and to contribute to the debate of enhanced interoperability of EODC. We demonstrate how standards can be used, profiled and enriched to pave the way to increased interoperability of EODC and can help delivering and leveraging the power of EO data building, efficient discovery, access and processing services.

A Novel Strategy for Very-Large-Scale Cash-Crop Mapping in the Context of Weather-Related Risk Assessment, Combining Global Satellite Multispectral Datasets, Environmental Constraints, and In Situ Acquisition of Geospatial Data.

Cash crops are agricultural crops intended to be sold for profit as opposed to subsistence crops, meant to support the producer, or to support livestock. Since cash crops are intended for future sale, they translate into large financial value when considered on a wide geographical scale, so their production directly involves financial risk. At a national level, extreme weather events including destructive rain or hail, as well as drought, can have a significant impact on the overall economic balance. It is thus important to map such crops in order to set up insurance and mitigation strategies. Using locally generated data—such as municipality-level records of crop seeding—for mapping purposes implies facing a series of issues like data availability, quality, homogeneity, etc. We thus opted for a different approach relying on global datasets. Global datasets ensure homogeneity and availability of data, although sometimes at the expense of precision and accuracy. A typical global approach makes use of spaceborne remote sensing, for which different land cover classification strategies are available in literature at different levels of cost and accuracy. We selected the optimal strategy in the perspective of a global processing chain. Thanks to a specifically developed strategy for fusing unsupervised classification results with environmental constraints and other geospatial inputs including ground-based data, we managed to obtain good classification results despite the constraints placed. The overall production process was composed using “good-enough" algorithms at each step, ensuring that the precision, accuracy, and data-hunger of each algorithm was commensurate to the precision, accuracy, and amount of data available. This paper describes the tailored strategy developed on the occasion as a cooperation among different groups with diverse backgrounds, a strategy which is believed to be profitably reusable in other, similar contexts. The paper presents the problem, the constraints and the adopted solutions; it then summarizes the main findings including that efforts and costs can be saved on the side of Earth Observation data processing when additional ground-based data are available to support the mapping task.

The Potential of Earth Observation for the Analysis of Cold Region Land Surface Dynamics in Europe—A Review

Cold regions affect global, regional and local climate; oftentimes they are relevant for water supply, host valuable ecosystems, and support human livelihood. They are thus eminently important for human society. In the context of ongoing climate change, monitoring and understanding cold region land surface dynamics is essential for environmental scientists, stakeholders and decision makers. However, the definition of cold regions remains inexplicit, and no up-to-date cold region maps or overarching spatial analyses exist. For example, Europe has densely populated cold regions, but hardly an article exists that provides a solid overview of Earth Observation (EO) based applications assessing cold region land surface dynamics in Europe. With this review article we aim at closing this gap by providing an overview of EO-based techniques for cold region observation in Europe, focusing on the dynamics of glaciers and snow. We present a novel spatial delineation of cold regions for Europe before analyzing the benefits and limitations of different EO sensor types and data processing methods for EO based cold region research. Furthermore, we identify research gaps and discuss challenges for future studies.

Big Data Management in Earth Observation: The German satellite data archive at the German Aerospace Center

The German Satellite Data Archive (D-SDA) at the German Aerospace Center (DLR) has been managing largevolume Earth-observation (EO) data in the context of EOmission payload ground segments (PGSs) for more than two decades. Hardware, data management, processing, user access, long-term preservation, and data exploitation expertise are under one roof and interact closely. Upcoming EO-mission PGSs benefit as much from the comprehensive expertise, close interaction, and integrated infrastructure as do in-house scientific application projects requiring access, processing, and archiving of large-volume EO data. Using a number of examples, we will demonstrate how EO data life cycles benefit from the proximity of data management and application scientists and from the extensive operational experience gathered over time.

Web Based Land Information System for Nintavur Divisional Secretariat Division (DSD), Sri Lanka

Nowadays, Web-based Land Information System is important to learn about the use of Geographic Information Systems and software for our profession. Modern technologies like geo-data processing, earth observation data processing and analysis are needed for Sri Lankan young researchers and students. This research work is dedicated to develop a land information system of Nintavur DSD. One task of the work is to establish a land information database which is based on an Open Source Geographic Information System. Work is starting with acquisition of existing geodata and information, qualification, maintenance, utilization and transferring of data from Nintavur DSD. The establishment of a land information system for Nintavur DSD will contribute to the National Land Information System of Sri Lanka. The study demonstrates that Land Information Systems can be developed using GIS software as e.g. QuantumGIS, and the Open Source database management system Postgre SQL /Post GIS and Arc GIS. This paper presents a framework for a Web-based GIS for Land Information. It focuses on the underlying concepts, theories and techniques for designing and implementing the conceptual Original Research Article Kaleel and Zahir; JGEESI, 5(4): 1-6, 2016; Article no.JGEESI.20748 2 framework that called WebGIS Land information. It consists of two main elements supporting the deliberative and analytic components of decision-making process. The deliberative part is based on the concept of argumentation maps. The analytic component consists of decision analysis methods. WebGIS Land information uses the server-side architecture approach to Web-based GIS. It employs HTML, CSS and JavaScript on the client-side and a combination of PHP scripting language, Map Server and a Postgres Sql database on the WebGIS Land Information.

Processing of earth observation big data: Challenges and countermeasures

Earth observation uses spaceborne sensors to obtain complete and systematic information about the earth. With the help of advances in spatial information technologies, the field of earth observation has entered the era of “big data”. By analyzing the entire process of earth observation data processing and associated algorithms, this paper concludes that the challenges facing earth observation “big data” processing are those of data-intensive computing. We suggest that to solve this problem, there should be research and innovation in the areas of system platforms, processing algorithms, and services models.

An automated and integrated framework for dust storm detection based on ogc web processing services

Abstract. Dust storms are known to have adverse effects on public health. Atmospheric dust loading is also one of the major uncertainties in global climatic modelling as it is known to have a significant impact on the radiation budget and atmospheric stability. The complexity of building scientific dust storm models is coupled with the scientific computation advancement, ongoing computing platform development, and the development of heterogeneous Earth Observation (EO) networks. It is a challenging task to develop an integrated and automated scheme for dust storm detection that combines Geo-Processing frameworks, scientific models and EO data together to enable the dust storm detection and tracking processes in a dynamic and timely manner. This study develops an automated and integrated framework for dust storm detection and tracking based on the Web Processing Services (WPS) initiated by Open Geospatial Consortium (OGC). The presented WPS framework consists of EO data retrieval components, dust storm detecting and tracking component, and service chain orchestration engine. The EO data processing component is implemented based on OPeNDAP standard. The dust storm detecting and tracking component combines three earth scientific models, which are SBDART model (for computing aerosol optical depth (AOT) of dust particles), WRF model (for simulating meteorological parameters) and HYSPLIT model (for simulating the dust storm transport processes). The service chain orchestration engine is implemented based on Business Process Execution Language for Web Service (BPEL4WS) using open-source software. The output results, including horizontal and vertical AOT distribution of dust particles as well as their transport paths, were represented using KML/XML and displayed in Google Earth. A serious dust storm, which occurred over East Asia from 26 to 28 Apr 2012, is used to test the applicability of the proposed WPS framework. Our aim here is to solve a specific instance of a complex EO data and scientific model integration problem by using a framework and scientific workflow approach together. The experimental result shows that this newly automated and integrated framework can be used to give advance near real-time warning of dust storms, for both environmental authorities and public. The methods presented in this paper might be also generalized to other types of Earth system models, leading to improved ease of use and flexibility.

Enabling the Use of Earth Observation Data for Integrated Water Resource Management in Africa with the Water Observation and Information System

The Water Observation and Information System (WOIS) is an open source software tool for monitoring, assessing and inventorying water resources in a cost-effective manner using Earth Observation (EO) data. The WOIS has been developed by, among others, the authors of this paper under the TIGER-NET project, which is a major component of the TIGER initiative of the European Space Agency (ESA) and whose main goal is to support the African Earth Observation Capacity for Water Resource Monitoring. TIGER-NET aims to support the satellite-based assessment and monitoring of water resources from watershed to cross-border basin levels through the provision of a free and powerful software package, with associated capacity building, to African authorities. More than 28 EO data processing solutions for water resource management tasks have been developed, in correspondence with the requirements of the participating key African water authorities, and demonstrated with dedicated case studies utilizing the software in operational scenarios. They cover a wide range of themes and information products, including basin-wide characterization of land and water resources, lake water quality monitoring, hydrological modeling and flood forecasting and mapping. For each monitoring task, step-by-step workflows were developed, which can either be adjusted by the user or largely automatized to feed into existing data streams and reporting schemes. The WOIS enables African water authorities to fully exploit the increasing EO capacity offered by current and upcoming generations of satellites, including the Sentinel missions.

Human Settlements: A Global Challenge for EO Data Processing and Interpretation

The availability of fine spatial resolution earth observation (EO) data has been always considered as a plus for human settlement monitoring from space. It calls, however, for new and efficient data processing tools, capable to manage huge data amounts and provide information for urban area monitoring, management, and protection. Issues related to data processing at the global level imply multiple-scale processing and a focused data interpretation approach, starting from human settlement delineation using spatial information and detailing down to material identification and object characterization using multispectral and/or hyperspectral data sets. This paper attempts to provide a consistent framework for information processing in urban remote sensing, stressing the need for a global approach able to exploit the detailed information available from EO data sets.

Cloud Computing Enabled Web Processing Service for Earth Observation Data Processing

The OpenGIS Web Processing Service (WPS) can process both simple and complex geospatial tasks including Earth Observation tasks. As the requirements of Earth Observation data, algorithms, calculation models, and daily life become increasingly complicated; WPS needs to provide high-performance service-oriented computing capability. This paper proposes a cloud computing enabled WPS framework for Earth Observation data processing. It consists of a client layer and a WPS layer, which further consists of a WPS server layer and a cloud computing layer. The cloud computing environment is based on the open-source software Apache Hadoop. The three layers of the proposed cloud computing enabled WPS are outlined, followed by a workflow that processes a user's task using these three layers. Then technological implementation details are explained. An experiment processing Moderate Resolution Imaging Spectroradiometer (MODIS) data shows that WPS can be enabled in a cloud computing environment.