Geospatial Raster Data Research Articles

ГЕОДЕЗІЯ, КАРТОГРАФІЯ І АЕРОФОТОЗНІМАННЯ

The aim of the work is to propose a method of creating a geographic information online resource for the management of Lisovohrynivetska UTC. To implement the tasks, a technological scheme was proposed, which consisted of 9 stages of work. The first stage involved the collection and analysis of disparate data in both vector and raster formats on the territory of the Lisovohrenivetska united territorial community. In the second stage, with the help of Global Mapper software, all vector data files in *.dxf and *.dmf formats, which were previously available, were converted to *.shp format for further processing in ArcGIS software. As a result of the conversion, graphic and attributive data were obtained in the required format and according to the layers they contain, the geodatabase with symbols according to the classifier was edited to create 1: 2000 scale plans. The next step was to unify the database of convertible files, as vector data was created with different construction of attribute tables. In addition, there is a need to enter vector data into the edited geospatial database. To do this, a ArcPy script was written that rearranges attribute tables and enters data into a geodatabase. Adjusted and populated the attribute database of vector objects for those columns where there was no information. The penultimate step was to develop the structure of the geoportal on the basis of ArcGIS-online to download the geodatabase to Lisovohrynivetska UTC on the server, to enable their external use with a unique login and password. The last step, after creating the structure of the geoportal, was to upload vector and raster geodata prepared by ArcGIS to the geoportal. As a result of the realization of the set purpose the technique of creating the geoinformation online resource for the management of the united territorial community is offered and described. During the implementation of the method the data of 24 disparate vector layers for the Lisogrynivtska community of Khmelnytsky region were processed and converted. Raster cartographic materials for UTC were collected and processed. The geodatabase according to the classifier for scale 1: 2000 is created. The structure of the geoportal based on the ArcGIS-online kernel with a connected map-base based on the online resource GoogleMaps, where all processed materials are downloaded, has been developed. The scientific novelty is to develop the concept of accumulation of heterogeneous vector and raster geospatial data in one geodatabase, by converting them into a specific format. Additional modules have been written in ArcPy to unify the database structure. Implemented geoinformation system is located on the geoportal and is designed for management decisions of community leaders. In addition, the created GIS can be used for land management and surveying work on community sites.

Evaluation of a NoSQL Database for Storing Big Geospatial Raster Data

Database systems capable of efficiently storing geospatial data are widespread. However, recent developments in earth observation systems, remote sensing, mobile mapping, and crowd sourcing lead to large amounts of geospatial mass data that can hardly be handled efficiently with the existing solutions. Especially large geospatial raster data require novel concepts for well-organized data storage. A concept for storage of large geospatially and temporally referenced image data using the NoSQL graph database system Neo4j as a research subject of the project “RiverView®” is introduced. New strategies and access structures have been developed to ensure the persistence and performant access to image data in Neo4j. These strategies are compared with the up-and download performance of the widespread Rasdaman array database system.

Multidimensional Visualization and Processing of Big Open Urban Geospatial Data on the Web

The focus of this research is addressing a subset of the geovisualization (i.e., geographic visualization) challenges identified in the literature, namely multidimensional vector and raster geospatial data visualization. Moreover, the work implements an approach for multidimensional raster geospatial data processing. The results of this research are provided through a geoportal comprised of multiple applications that are related to 3D visualization of cities, ground deformation, land use and land cover and mobility. In a subset of the applications, the datasets handled are considered to be large in volume. The geospatial data were visualized on dynamic and interactive virtual globes to enable visual exploration. The geoportal is available on the web to enable cross-platform access to it. Furthermore, the geoportal was developed employing open standards, free and open source software (FOSS) and open data, most importantly to ensure interoperability and reduce the barriers to access it. The geoportal brings together various datasets, different both in terms of context and format employing numerous technologies. As a result, the existing web technologies for geovisualization and geospatial data processing were examined and exemplary and innovative software was developed to extend the state of the art.

A geostatistical approach for multi-source data fusion to predict water table depth

Accurate water table depth mapping is important for water management and activity planning. The joint use of exhausted geospatial raster data with sparse field measurements could improve predictions. The aim of this work was to fuse different support data, collected with remote sensors, with point soil field observations to improve water table depth prediction. A method for multi-source data fusion is described in detail, based on multivariate geostatistics and exemplified with a case study in a conservation area of 5700 ha in the state of São Paulo, Brazil. TanDEM-X digital surface model with 90 m resolution and SAFER (Simple Algorithm for Evapotranspiration Retrieving) data calculated from Sentinel-2 images with 20 m resolution, were jointly used with water table depth and soil physical variables measured at 56 locations to predict water table depth in two hydrological years (2015–16 and 2016–17). Data were transformed to normal distributions using the Gaussian anamorphosis approach. A Linear Model of Coregionalization (LMC), calculated for all direct and cross-variograms of the eleven variables of study, was regularized at block support for multi-collocated block cokriging predictions. Support change correction was made to reduce punctual variance to block variances. Univariate and multivariate geostatistical interpolation methods were compared through cross validation. The uncertainty associated to the water table depths estimated by multivariate approach was lower than those by the univariate approach. Moreover, multivariate predictions incorporated the influences induced by local relief, vegetation and soil properties. Confidence interval maps, presented as uncertainty measure, reveal areas with higher and lower precision of groundwater level prediction that could be effectively used as support in land use management.

OCTOPUS: an open cosmogenic isotope and luminescence database

Abstract. We present a database of cosmogenic radionuclide and luminescence measurements in fluvial sediment. With support from the Australian National Data Service (ANDS) we have built infrastructure for hosting and maintaining the data at the University of Wollongong and making this available to the research community via an Open Geospatial Consortium (OGC)-compliant web service. The cosmogenic radionuclide (CRN) part of the database consists of 10Be and 26Al measurements in modern fluvial sediment samples from across the globe, along with ancillary geospatial vector and raster layers, including sample site, basin outline, digital elevation model, gradient raster, flow-direction and flow-accumulation rasters, atmospheric pressure raster, and CRN production scaling and topographic shielding factor rasters. Sample metadata are comprehensive and include all necessary information for the recalculation of denudation rates using CAIRN, an open-source program for calculating basin-wide denudation rates from 10Be and 26Al data. Further all data have been recalculated and harmonised using the same program. The luminescence part of the database consists of thermoluminescence (TL) and optically stimulated luminescence (OSL) measurements in fluvial sediment samples from stratigraphic sections and sediment cores from across the Australian continent and includes ancillary vector and raster geospatial data. The database can be interrogated and downloaded via a custom-built web map service. More advanced interrogation and exporting to various data formats, including the ESRI Shapefile and Google Earth's KML, is also possible via the Web Feature Service (WFS) capability running on the OCTOPUS server. Use of open standards also ensures that data layers are visible to other OGC-compliant data-sharing services. OCTOPUS and its associated data curation framework provide the opportunity for researchers to reuse previously published but otherwise unusable CRN and luminescence data. This delivers the potential to harness old but valuable data that would otherwise be lost to the research community. OCTOPUS can be accessed at https://earth.uow.edu.au (last access: 28 November 2018). The individual data collections can also be accessed via the following DOIs: https://doi.org/10.4225/48/5a8367feac9b2 (CRN International), https://doi.org/10.4225/48/5a836cdfac9b5 (CRN Australia), and https://doi.org/10.4225/48/5a836db1ac9b6 (OSL & TL Australia).

A hierarchical indexing strategy for optimizing Apache Spark with HDFS to efficiently query big geospatial raster data

ABSTRACTEarth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.

Evaluating the Open Source Data Containers for Handling Big Geospatial Raster Data

Big geospatial raster data pose a grand challenge to data management technologies for effective big data query and processing. To address these challenges, various big data container solutions have been developed or enhanced to facilitate data storage, retrieval, and analysis. Data containers were also developed or enhanced to handle geospatial data. For example, Rasdaman was developed to handle raster data and GeoSpark/SpatialHadoop were enhanced from Spark/Hadoop to handle vector data. However, there are few studies to systematically compare and evaluate the features and performances of these popular data containers. This paper provides a comprehensive evaluation of six popular data containers (i.e., Rasdaman, SciDB, Spark, ClimateSpark, Hive, and MongoDB) for handling multi-dimensional, array-based geospatial raster datasets. Their architectures, technologies, capabilities, and performance are compared and evaluated from two perspectives: (a) system design and architecture (distributed architecture, logical data model, physical data model, and data operations); and (b) practical use experience and performance (data preprocessing, data uploading, query speed, and resource consumption). Four major conclusions are offered: (1) no data containers, except ClimateSpark, have good support for the HDF data format used in this paper, requiring time- and resource-consuming data preprocessing to load data; (2) SciDB, Rasdaman, and MongoDB handle small/mediate volumes of data query well, whereas Spark and ClimateSpark can handle large volumes of data with stable resource consumption; (3) SciDB and Rasdaman provide mature array-based data operation and analytical functions, while the others lack these functions for users; and (4) SciDB, Spark, and Hive have better support of user defined functions (UDFs) to extend the system capability.

THE DESIGN OF A HIGH PERFORMANCE EARTH IMAGERY AND RASTER DATA MANAGEMENT AND PROCESSING PLATFORM

This paper summarizes the general requirements and specific characteristics of both geospatial raster database management system and raster data processing platform from a domain-specific perspective as well as from a computing point of view. It also discusses the need of tight integration between the database system and the processing system. These requirements resulted in Oracle Spatial GeoRaster, a global scale and high performance earth imagery and raster data management and processing platform. The rationale, design, implementation, and benefits of Oracle Spatial GeoRaster are described. Basically, as a database management system, GeoRaster defines an integrated raster data model, supports image compression, data manipulation, general and spatial indices, content and context based queries and updates, versioning, concurrency, security, replication, standby, backup and recovery, multitenancy, and ETL. It provides high scalability using computer and storage clustering. As a raster data processing platform, GeoRaster provides basic operations, image processing, raster analytics, and data distribution featuring high performance computing (HPC). Specifically, HPC features include locality computing, concurrent processing, parallel processing, and in-memory computing. In addition, the APIs and the plug-in architecture are discussed.

Towards a voxel-based geographic automata for the simulation of geospatial processes

Many geographic processes evolve in a three dimensional space and time continuum. However, when they are represented with the aid of geographic information systems (GIS) or geosimulation models they are modelled in a framework of two-dimensional space with an added temporal component. The objective of this study is to propose the design and implementation of voxel-based automata as a methodological approach for representing spatial processes evolving in the four-dimensional (4D) space–time domain. Similar to geographic automata models which are developed to capture and forecast geospatial processes that change in a two-dimensional spatial framework using cells (raster geospatial data), voxel automata rely on the automata theory and use three-dimensional volumetric units (voxels). Transition rules have been developed to represent various spatial processes which range from the movement of an object in 3D to the diffusion of airborne particles and landslide simulation. In addition, the proposed 4D models demonstrate that complex processes can be readily reproduced from simple transition functions without complex methodological approaches. The voxel-based automata approach provides a unique basis to model geospatial processes in 4D for the purpose of improving representation, analysis and understanding their spatiotemporal dynamics. This study contributes to the advancement of the concepts and framework of 4D GIS.

A Diskless Checkpointing Algorithm for Cluster Architectures Applied to Geospatial Raster Data Processing

In recent years, due to the increasing calculation demands for the massive spatial data analysis, the parallel computing based on high-performance computers has become an inevitable trend of geospatial raster data processing, such as digital terrain analysis (DTA for short), remote sensing interpretation and digital soil mapping. A key problem is how to design a fault-tolerant software to enhance the stability and robustness of scientific application. This paper presents an approach of failure recovery for distributed memory parallel computing. Furthermore, we adopt the master/slave programming model and present a framework of redundant master mode, which the failure occurring on the master node could not lead to a breakdown of the whole system. This approach schedules the failing task by dividing all the failing data into several partitions according to the calculating scale of failure. By means of the Fault-Tolerant Granularity Model, the scheduling algorithm can assign the failing task dynamically. Finally, taking example of digital terrain analysis, two experiments are discussed that based on the data size and the number of failures. Simulation results indicate that the proposed scheduling algorithm based on Fault-Tolerant Granularity Model achieves lower fault tolerance overhead than the rollback recovery scheme when several processors fail simultaneously.

Web based geoprocessing tool for coverage data handling

Abstract. With the advancements in GIS technologies and extensive use of OGC Web Services, geospatial resources and services are becoming progressively copious and convenient over the network. The application of OGC WCS (Web Coverage Service) and WFS (Web Feature Service) standards for geospatial raster and vector data has resulted in an opulent pool of interoperable geodata resources waiting to be used for analytical or modelling purposes. The issue of availing geospatial data processing with the aid of standardised web services was attended to by the OGC WPS (Web Processing Service) 1.0.0 specifications (Schut, 2007) which elucidate WPS as a standard interface which serves for the promulgation of geo-processes and consumption of those processes by the clients. This paper outlines the design and implementation of a geo-processing tool utilizing coverage data. The geo-process selected for application is the calculation of Normalized Difference Vegetative Index (NDVI), one of the globally used indices for vegetation cover monitoring. The system is realised using the Geospatial Data Abstraction Library (GDAL) and Python. The tool accesses the WCS server using the parameters defined in the XML request. The geo-process upon execution, performs the computations over the coverage data and generates the NDVI output. Since open source technology and standards are being used more often, especially in the field of scientific research, so our implementation is also built by using open source tools only.

PRPL 2.0: Improving the Parallel Raster Processing Library

AbstractThis article presents an improved parallel Raster Processing Library – pRPL version 2.0. Since the release of version 1.0, a series of modifications has been made in pRPL to improve its usability, flexibility, and performance. While retaining some of the key features of pRPL, the new version has gained several new features: (1) a new DataManager class has been added for integrated data management, and to facilitate data decomposition, assignment mapping, data distribution, Transition execution, and load‐balancing; (2) a GDAL‐based raster data I/O mechanism has been added to support various geospatial raster data formats, and provide centralized and pseudo parallel I/O modes; and (3) a static load‐balancing mode and a dynamic load‐balancing mode using the task‐farming technique are provided. A parallel zonal statistics tool and a parallel Cellular Automata model were developed to demonstrate the usability and performance of pRPL 2.0. The experiments using the California datasets showed that the performance altered when different pRPL options (i.e. load‐balancing mode, I/O mode and writer mode) were used for different algorithms, datasets, and varying numbers of processes.

How to Apply the Geospatial Data Abstraction Library (GDAL) Properly to Parallel Geospatial Raster I/O?

AbstractInput/output (I/O) of geospatial raster data often becomes the bottleneck of parallel geospatial processing due to the large data size and diverse formats of raster data. The open‐source Geospatial Data Abstraction Library (GDAL), which has been widely used to access diverse formats of geospatial raster data, has been applied recently to parallel geospatial raster processing. This article first explores the efficiency and feasibility of parallel raster I/O using GDAL under three common ways of domain decomposition: row‐wise, column‐wise, and block‐wise. Experimental results show that parallel raster I/O using GDAL under column‐wise or block‐wise domain decomposition is highly inefficient and cannot achieve correct output, although GDAL performs well under row‐wise domain decomposition. The reasons for this problem with GDAL are then analyzed and a two‐phase I/O strategy is proposed, designed to overcome this problem. A data redistribution module based on the proposed I/O strategy is implemented for GDAL using a message‐passing‐interface (MPI) programming model. Experimental results show that the data redistribution module is effective.

A Program for Handling Map Projections of Small Scale Geospatial Raster Data

Scientists routinely accomplish small-scale geospatial modeling using raster datasets of global extent. Such use often requires the projection of global raster datasets onto a map or the reprojection from a given map projection associated with a dataset. The distortion characteristics of these projection transformations can have significant effects on modeling results. Distortions associated with the reprojection of global data are generally greater than distortions associated with reprojections of larger-scale, localized areas. The accuracy of areas in projected raster datasets of global extent is dependent on resolution. To address these problems of projection and the associated resampling that accompanies it, methods for framing the transformation space, direct point-to-point transformations rather than gridded transformation spaces, a solution to the wrap-around problem, and an approach to alternative resampling methods are presented. The implementations of these methods are provided in an open source software package called MapImage (or mapIMG, for short), which is designed to function on a variety of computer architectures.

Land Cover and Land Use Classification with TWOPAC: towards Automated Processing for Pixel- and Object-Based Image Classification

We present a novel and innovative automated processing environment for the derivation of land cover (LC) and land use (LU) information. This processing framework named TWOPAC (TWinned Object and Pixel based Automated classification Chain) enables the standardized, independent, user-friendly, and comparable derivation of LC and LU information, with minimized manual classification labor. TWOPAC allows classification of multi-spectral and multi-temporal remote sensing imagery from different sensor types. TWOPAC enables not only pixel-based classification, but also allows classification based on object-based characteristics. Classification is based on a Decision Tree approach (DT) for which the well-known C5.0 code has been implemented, which builds decision trees based on the concept of information entropy. TWOPAC enables automatic generation of the decision tree classifier based on a C5.0-retrieved ascii-file, as well as fully automatic validation of the classification output via sample based accuracy assessment.Envisaging the automated generation of standardized land cover products, as well as area-wide classification of large amounts of data in preferably a short processing time, standardized interfaces for process control, Web Processing Services (WPS), as introduced by the Open Geospatial Consortium (OGC), are utilized. TWOPAC’s functionality to process geospatial raster or vector data via web resources (server, network) enables TWOPAC’s usability independent of any commercial client or desktop software and allows for large scale data processing on servers. Furthermore, the components of TWOPAC were built-up using open source code components and are implemented as a plug-in for Quantum GIS software for easy handling of the classification process from the user’s perspective.