Use Of Datasets Research Articles

Development and Validation of LiDAR Sensor Simulators Based on Parallel Raycasting.

Three-dimensional (3D) imaging technologies have been increasingly explored in academia and the industrial sector, especially the ones yielding point clouds. However, obtaining these data can still be expensive and time-consuming, reducing the efficiency of procedures dependent on large datasets, such as the generation of data for machine learning training, forest canopy calculation, and subsea survey. A trending solution is developing simulators for imaging systems, performing the virtual scanning of the digital world, and generating synthetic point clouds from the targets. This work presents a guideline for the development of modular Light Detection and Ranging (LiDAR) system simulators based on parallel raycasting algorithms, with its sensor modeled by metrological parameters and error models. A procedure for calibrating the sensor is also presented, based on comparing with the measurements made by a commercial LiDAR sensor. The sensor simulator developed as a case study resulted in a robust generation of synthetic point clouds in different scenarios, enabling the creation of datasets for use in concept tests, combining real and virtual data, among other applications.

What have biological records ever done for us? A systematic scoping review

Biological records provide biodiversity information over large spatial and temporal scales. Our systematic scoping review of biological records from the well-recorded region of the United Kingdom (UK) and Ireland revealed that over half of all studies using biological records were studying species distributions (134 of 253 studies) and/or temporal trends (139 of 253 studies). A minority of studies (61 of 253) focused on methodological questions, while most studies used biological records with existing methods as tools for answering biological and ecological questions. However, only 31 of 253 studies tested models using independent data. Most studies (154 of 253) integrated multiple biological records datasets, showing that biological records hold a largely untapped potential for independently testing conclusions by withholding some of those datasets for use as independent test data. Our results provide guidance for data providers and researchers interested in more effectively collecting and using biological records.

The content and accuracy of the CORINE Land Cover dataset for Norway

The CORINE Land Cover dataset for Norway for the reference year 2018 (CLC2018) was compared to detailed national land cover and land use data. This allowed us to describe the thematic composition of the CLC-polygons and aggregate the information into statistical profiles for each CLC-class. We compared the results to the class definitions found in the CLC mapping instructions, while considering the generalization and minimal mapping units required for CLC. The study showed that CLC2018 in general complied with the definitions. Nonconformities were mainly found for detailed and (in a Norwegian context) marginal classes. The classification can still be improved by complementing visual interpretation with classification based on the statistical profile of each polygon when detailed land use and land cover information is available. The use of auxiliary information at the polygon level can thus provide a better, thematically more accurate CLC dataset for use in European land monitoring.

Genomes OnLine Database (GOLD) v.8: overview and updates.

The Genomes OnLine Database (GOLD) (https://gold.jgi.doe.gov/) is a manually curated, daily updated collection of genome projects and their metadata accumulated from around the world. The current version of the database includes over 1.17 million entries organized broadly into Studies (45 770), Organisms (387 382) or Biosamples (101 207), Sequencing Projects (355 364) and Analysis Projects (283 481). These four levels contain over 600 metadata fields, which includes 76 controlled vocabulary (CV) tables containing 3873 terms. GOLD provides an interactive web user interface for browsing and searching by a wide range of project and metadata fields. Users can enter details about their own projects in GOLD, which acts as a gatekeeper to ensure that metadata is accurately documented before submitting sequence information to the Integrated Microbial Genomes (IMG) system for analysis. In order to maintain a reference dataset for use by members of the scientific community, GOLD also imports projects from public repositories such as GenBank and SRA. The current status of the database, along with recent updates and improvements are described in this manuscript.

An Architectural Multi-Agent System for a Pavement Monitoring System with Pothole Recognition in UAV Images.

In recent years, maintenance work on public transport routes has drastically decreased in many countries due to difficult economic situations. The various studies that have been conducted by groups of drivers and groups related to road safety concluded that accidents are increasing due to the poor conditions of road surfaces, even affecting the condition of vehicles through costly breakdowns. Currently, the processes of detecting any type of damage to a road are carried out manually or are based on the use of a road vehicle, which incurs a high labor cost. To solve this problem, many research centers are investigating image processing techniques to identify poor-condition road areas using deep learning algorithms. The main objective of this work is to design of a distributed platform that allows the detection of damage to transport routes using drones and to provide the results of the most important classifiers. A case study is presented using a multi-agent system based on PANGEA that coordinates the different parts of the architecture using techniques based on ubiquitous computing. The results obtained by means of the customization of the You Only Look Once (YOLO) v4 classifier are promising, reaching an accuracy of more than 95%. The images used have been published in a dataset for use by the scientific community.

Development of pacific exposure dataset for use in catastrophe risk assessment

An exposure dataset is one of the critical components of catastrophe risk modelling. It can also be one of the most difficult to create. When small numbers of buildings are involved, less than a thousand or so, it may be practicable to view each building and to acquire all the necessary attributes with a reasonable level of confidence. For regional or national-scale projects, however, the task is next to impossible. Ideally, for each building within the modelling areas we would know the location and value, and have enough structural information to underpin the assignment of vulnerability functions for each of several hazards. Such a dataset does not exist in the Pacific region, though various exposure databases have been developed in the past couple of decades. Instead, we have developed a systematic approach to populate and maintain a somewhat less rigorous, but practicable, exposure dataset for use in catastrophe risk modelling. In this paper, a brief review of previous development of exposure databases in the Pacific region is presented, followed by an overview of each class of available data that is used to develop the exposure dataset. Next, the methodologies adopted are illustrated, and application to a test case in Tanna Island, Vanuatu is described. Finally, the proposed exposure dataset development is discussed.

PODPAC: open-source Python software for enabling harmonized, plug-and-play processing of disparate earth observation data sets and seamless transition onto the serverless cloud by earth scientists

In this paper, we present the Pipeline for Observational Data Processing, Analysis, and Collaboration (PODPAC) software. PODPAC is an open-source Python library designed to enable widespread exploitation of NASA earth science data by enabling multi-scale and multi-windowed access, exploration, and integration of available earth science datasets to support analysis and analytics; automatic accounting for geospatial data formats, projections, and resolutions; simplified implementation and parallelization of geospatial data processing routines; standardized sharing of data and algorithms; and seamless transition of algorithms and data products from local development to distributed, serverless processing on commercial cloud computing environments. We describe the key elements of PODPAC’s architecture, including Nodes for unified encapsulation of disparate scientific data sources; Algorithms for plug-and-play processing and harmonization of multiple data source Nodes; and Lambda functions for serverless execution and sharing of new data products via the cloud. We provide an overview of our open-source code implementation and testing process for development and deployment of PODPAC. We describe our interactive, JupyterLab-based end-user documentation including quick-start examples and detailed use case studies. We conclude with examples of PODPAC’s application to: encapsulate data sources available on Amazon Web Services (AWS) Open Data repository; harmonize processing of multiple earth science data sets for downscaling of NASA Soil Moisture Active Passive (SMAP) soil moisture data; and deploy a serverless SMAP-based drought monitoring application for use access from mobile devices. We postulate that PODPAC will also be an effective tool for wrangling and standardizing massive earth science data sets for use in model training for machine learning applications.

Lessons from a Marine Spatial Planning data management process for Ireland

ABSTRACT This paper presents a framework containing ten components to deliver a data management process for the storage and management of data used for Marine Spatial Planning (MSP) in Ireland. The work includes a data process flow and a recommended solution architecture. The architecture includes a central data catalogue and a spatial storage system. The components of the process are presented to maximise the reuse potential of any dataset within an MSP context. The terms ‘Suitability’ and ‘Readiness’ in the MSP context are offered as both formal and considered assessments of data, as is the applicability of a data stewardship maturity matrix. How data contained in such a storage system can be published externally to potential consumers of these data is also explored. The process presents a means of managing data and metadata to ensure data lineage is optimised by carrying information about the origin of and the processing applied to the data; to evaluate the quality and relevance of geospatial datasets for use in MSP decisions in Ireland. The process was piloted in the National Marine Planning Framework for Ireland in the development of draft map products; feedback from the public consultation is ongoing and not presented.

Performance of various gridded temperature and precipitation datasets over Northwest Himalayan Region

This study evaluated the performance of 07 gridded datasets viz. Asian Precipitation Highly-resolved Observational Data Integration towards Evaluation of Water Resources (APHRODITE), Climate Research Unit Time-Series (CRU-TS), University of Delaware (UDEL), Tropical rainfall Measurement Mission (TRMM)/ TMPA (TRMM Multi-Satellite Precipitation Analysis), Global Precipitation Climatology Centre (GPCC), Princeton Global Forcings Dataset (PGF), and European Reanalysis Interim (ERA-I) in capturing the amount, seasonality and trend of precipitation over different climatic zones of Northwestern Himalaya (NWH) i.e. Lower Himalaya (LH), Greater Himalaya (GH) and Karakoram Himalaya (KH). A similar comparison was also done for the temperature data but only with 05 datasets, viz. APHRODITE, CRU-TS, PGF, UDEL and ERA-I since TMPA and GPCC are precipitation datasets only. This study is a maiden attempt where in situ observation includes the data from elevations above 5000 m amsl (07 observatories) in NWH (Indian sub-region). Results reveal that for precipitation over NWH; ERA-I, GPCC, and TMPA/TRMM were found to be quite reliable datasets. For temperature, all datasets performed quite well but CRU-TS and ERA-I provided more reliable estimates. The mean absolute error ranged from 13.5 mm/month to 150.7 mm/month for precipitation and 0.75°C/month to 9.9°C/month for temperature. High values of the errors underpin the need for bias correction. On the basis of this analysis, monthly correction factors for wintertime temperature and precipitation have also been suggested for each dataset which when multiplied with corresponding datasets would result in closely approximated values for the area of interest. These results can serve as a guide for bias correction and selection of appropriate gridded datasets for use in studies pertaining to hydrological modeling over NWH.

Compilation of training datasets for use of convolutional neural networks supporting automatic inspection processes in industry 4.0 based electronic manufacturing

Abstract. Ensuring the highest quality standards at competitive prices is one of the greatest challenges in the manufacture of electronic products. The identification of flaws has the uppermost priority in the field of automotive electronics, particularly as a failure within this field can result in damages and fatalities. During assembling and soldering of printed circuit boards (PCBs) the circuit carriers can be subject to errors. Hence, automatic optical inspection (AOI) systems are used for real-time detection of visible flaws and defects in production. This article introduces an application strategy for combining a deep learning concept with an optical inspection system based on image processing. Above all, the target is to reduce the risk of error slip through a second inspection. The concept is to have the inspection results additionally evaluated by a convolutional neural network. For this purpose, different training datasets for the deep learning procedures are examined and their effects on the classification accuracy for defect identification are assessed. Furthermore, a suitable compilation of image datasets is elaborated, which ensures the best possible error identification on solder joints of electrical assemblies. With the help of the results, convolutional neural networks can achieve a good recognition performance, so that these can support the automatic optical inspection in a profitable manner. Further research aims at integrating the concept in a fully automated way into the production process in order to decide on the product quality autonomously without human interference.

A time-scale modification dataset with subjective quality labels.

Time Scale Modification (TSM) is a well-researched field; however, no effective objective measure of quality exists. This paper details the creation, subjective evaluation, and analysis of a dataset for use in the development of an objective measure of quality for TSM. Comprised of two parts, the training component contains 88 source files processed using six TSM methods at 10 time scales, while the testing component contains 20 source files processed using three additional methods at four time scales. The source material contains speech, solo harmonic and percussive instruments, sound effects, and a range of music genres. Ratings (42 529) were collected from 633 sessions using laboratory and remote collection methods. Analysis of results shows no correlation between age and quality of rating; expert and non-expert listeners to be equivalent; minor differences between participants with and without hearing issues; and minimal differences between testing modalities. A comparison of published objective measures and subjective scores shows the objective measures to be poor indicators of subjective quality. Initial results for a retrained objective measure of quality are presented with results approaching average root mean squared error loss and Pearson correlation values of subjective sessions. The labeled dataset is available at http://ieee-dataport.org/1987.

Contextualizing Well-Being for Entrepreneurship

Entrepreneurship is important in economic growth and development. This study explores the likelihood that societal-level well-being and country-level self-expression values positively influence individual entrepreneurship across countries. Self-expression values mediate and reinforce the effect of societal well-being on entrepreneurship. Well-being is not simply an individual-level expression of positive emotions or an individual’s accumulated human capital alone. It is also a country’s stock of psychological as well as social resource and a macrolevel culture-specific emotion supporting entrepreneurship. This study provides a multidimensional approach to exploring the effects of societal well-being and country-level self-expression values on entrepreneurship. The proposed conceptual model uses three theoretical propositions to delineate an indirect effect of societal well-being mediated through country-level self-expression values. The study also compiles measures of societal well-being terms from secondary data sets for use in cross-country comparative empirical research into entrepreneurship.

Introducing Spatially Distributed Fire Danger from Earth Observations (FDEO) Using Satellite-Based Data in the Contiguous United States

Wildfire danger assessment is essential for operational allocation of fire management resources; with longer lead prediction, the more efficiently can resources be allocated regionally. Traditional studies focus on meteorological forecasts and fire danger index models (e.g., National Fire Danger Rating System—NFDRS) for predicting fire danger. Meteorological forecasts, however, lose accuracy beyond ~10 days; as such, there is no quantifiable method for predicting fire danger beyond 10 days. While some recent studies have statistically related hydrologic parameters and past wildfire area burned or occurrence to fire, no study has used these parameters to develop a monthly spatially distributed predictive model in the contiguous United States. Thus, the objective of this study is to introduce Fire Danger from Earth Observations (FDEO), which uses satellite data over the contiguous United States (CONUS) to enable two-month lead time prediction of wildfire danger, a sufficient lead time for planning purposes and relocating resources. In this study, we use satellite observations of land cover type, vapor pressure deficit, surface soil moisture, and the enhanced vegetation index, together with the United States Forest Service (USFS) verified and validated fire database (FPA) to develop spatially gridded probabilistic predictions of fire danger, defined as expected area burned as a deviation from “normal”. The results show that the model predicts spatial patterns of fire danger with 52% overall accuracy over the 2004–2013 record, and up to 75% overall accuracy during the fire season. Overall accuracy is defined as number of pixels with correctly predicted fire probability classes divided by the total number of the studied pixels. This overall accuracy is the first quantified result of two-month lead prediction of fire danger and demonstrates the potential utility of using diverse observational data sets for use in operational fire management resource allocation in the CONUS.

Global Estimates of Reach‐Level Bankfull River Width Leveraging Big Data Geospatial Analysis

AbstractRecent progress in remote sensing has snapshotted unprecedented numbers of river planform geometry, providing opportunity to revisit the oversimplified channel shape parameterizations in global hydrologic models. This study leveraged two recent Landsat‐derived global river width databases and created a reach‐level width dataset to measure the validity of model parameterizations at ~1.6 million kilometers of rivers in length. By showing state‐of‐the‐art parameterization schemes only capture 30–40% of the width variance globally, we developed a machine learning (ML) approach surveying 16 environmental covariates, which considerably improved the predictive power (R2 = 0.81 and 0.77 for two testing cases). Beyond the commonly discussed upstream basin conditions, ML revealed that local physiographic factors and human interference are also important covariates for width variability. Finally, we applied the ML model to estimate bankfull river width, creating a new reach‐level dataset for use in global hydrodynamic modeling.

CAPTCHA Image Generation: Two-Step Style-Transfer Learning in Deep Neural Networks.

Mobile devices such as sensors are used to connect to the Internet and provide services to users. Web services are vulnerable to automated attacks, which can restrict mobile devices from accessing websites. To prevent such automated attacks, CAPTCHAs are widely used as a security solution. However, when a high level of distortion has been applied to a CAPTCHA to make it resistant to automated attacks, the CAPTCHA becomes difficult for a human to recognize. In this work, we propose a method for generating a CAPTCHA image that will resist recognition by machines while maintaining its recognizability to humans. The method utilizes the style transfer method, and creates a new image, called a style-plugged-CAPTCHA image, by incorporating the styles of other images while keeping the content of the original CAPTCHA. In our experiment, we used the TensorFlow machine learning library and six CAPTCHA datasets in use on actual websites. The experimental results show that the proposed scheme reduces the rate of recognition by the DeCAPTCHA system to 3.5% and 3.2% using one style image and two style images, respectively, while maintaining recognizability by humans.

Multiscale Laboratory Study and Numerical Analysis of Water‐Weakening Effect on Shale

Water‐weakening effect is one of the most important factors inducing large deformation and stability problems of shale strata in the Huangjiazhai Tunnel. The influence of water on shale with various water contents and its mechanism still needs more study. In this paper, the X‐ray diffraction and water absorption test were first conducted to determine the influence of mineral components on water‐rock interaction. Then, a series of rock mechanical tests, including uniaxial compressive test, Brazilian disk test, and triaxial compressive test, were carried out on shale under dried condition, wetted condition with various water contents, and saturated condition. For the uniaxial compression, tension, and triaxial compression, the softening coefficient could reach 0.38, 0.63, and 0.52, respectively. Based on the macroscale experiments, a numerical case study of tunnel excavation was investigated to evaluate the water effect. Finally, with the aid of scanning electron microscope and failure morphology, the mechanism of water‐weakening effect on shale was discussed from microscopic and macroscopic perspectives. The thickness of the double layer increases with the increase of water immersion time. This paper provides a set of useful data for reference in construction in shale strata, and a methodology to evaluate water‐weakening effect through laboratory and numerical modelling approaches.

WasNet: A Neural Network-Based Garbage Collection Management System

In response to the increasing pollution caused by unseparated garbage, classification systems for garbage separation have become very popular. First, we constructed a complex data augmentation combination for model training. Second, we designed a novel lightweight neural network garbage classification system called WasNet. This proposed network's 1.5 million parameters on the ImageNet dataset are one-half of mainstream neural networks, while at 3 million floating point operations per second (FLOPs) it is one third of mainstream neural networks that have obtained the best performance among known lightweight neural networks. The accuracy on the ImageNet data set is 64.5%, on the Garbage Classification dataset it is 82.5%, and on the TrashNet dataset it is 96.10%. Furthermore, we transplanted the model to the hardware platform and assembled an intelligent trash can; we developed a garbage recognition application to facilitate users to directly identify and receive platform information; we built a visualization and decision support platform to help managers monitor traffic in real time. We combined the intelligent trash can, application, visualization and decision-making platform into a system, which is the most complete and effective system among the known research works. The results of the test we conducted on our platform using our extended dataset showed that our scheme is very reliable. At the same time, we also open source our extended datasets for use by other researchers.

Cross-docking benchmark for automated pose and ranking prediction of ligand binding.

Significant efforts have been devoted in the last decade to improving molecular docking techniques to predict both accurate binding poses and ranking affinities. Some shortcomings in the field are the limited number of standard methods for measuring docking success and the availability of widely accepted standard data sets for use as benchmarks in comparing different docking algorithms throughout the field. In order to address these issues, we have created a Cross-Docking Benchmark server. The server is a versatile cross-docking data set containing 4,399 protein-ligand complexes across 95 protein targets intended to serve as benchmark set and gold standard for state-of-the-art pose and ranking prediction in easy, medium, hard, or very hard docking targets. The benchmark along with a customizable cross-docking data set generation tool is available at http://disco.csb.pitt.edu. We further demonstrate the potential uses of the server in questions outside of basic benchmarking such as the selection of the ideal docking reference structure.

Mobile ground‐based SMART radar observations and wind retrievals during the landfall of Hurricane Harvey (2017)

AbstractThis article describes a unique ground‐based weather radar dataset collected during the landfall of Hurricane Harvey (2017) along the United States Gulf Coast, documents the wind retrievals conducted with this dataset, and reports on the location of the archive of the data and wind retrievals so that others may gain access and use of these data. Datasets from C‐band dual‐polarimetric Shared Mobile Atmospheric Research and Teaching (SMART) radar and the United States National Weather Service WSR‐88D in Corpus Christi, Texas, are presented, along with dual‐Doppler analyses before and during Harvey's landfall on the United States Gulf Coast. The quality assurance and dual‐Doppler wind synthesis procedures are detailed. Nearly 8 hr of dual‐Doppler wind analyses were constructed, providing an unprecedented dataset for use in understanding hurricane dynamics at landfall and validating numerical simulations of Hurricane Harvey. In addition, raw, dual‐polarization data not used in the dual‐Doppler syntheses, but included in each radar dataset, are also summarized.

Comparison and Assessment of Regional and Global Land Cover Datasets for Use in CLASS over Canada

Global land cover information is required to initialize land surface and Earth system models. In recent years, new land cover (LC) datasets at finer spatial resolutions have become available while those currently implemented in most models are outdated. This study assesses the applicability of the Climate Change Initiative (CCI) LC product for use in the Canadian Land Surface Scheme (CLASS) through comparison with finer resolution datasets over Canada, assisted with reference sample data and a vegetation continuous field tree cover fraction dataset. The results show that in comparison with the finer resolution maps over Canada, the 300 m CCI product provides much improved LC distribution over that from the 1 km GLC2000 dataset currently used to provide initial surface conditions in CLASS. However, the CCI dataset appears to overestimate needleleaf forest cover especially in the taiga-tundra transition zone of northwestern Canada. This may have partly resulted from limited availability of clear sky MEdium Resolution Imaging Spectrometer (MERIS) images used to generate the CCI classification maps due to the long snow cover season in Canada. In addition, changes based on the CCI time series are not always consistent with those from the MODIS or a Landsat-based forest cover change dataset, especially prior to 2003 when only coarse spatial resolution satellite data were available for change detection in the CCI product. It will be helpful for application in global simulations to determine whether these results also apply to other regions with similar landscapes, such as Eurasia. Nevertheless, the detailed LC classes and finer spatial resolution in the CCI dataset provide an improved reference map for use in land surface models in Canada. The results also suggest that uncertainties in the current cross-walking tables are a major source of the often large differences in the plant functional types (PFT) maps, and should be an area of focus in future work.