Data Discovery Process Research Articles

Enriching Earth observation datasets through semantics for climate change applications: The EIFFEL ontology.

Earth Observation (EO) datasets have become vital for decision support applications, particularly from open satellite portals that provide extensive historical datasets. These datasets can be integrated with in-situ data to power artificial intelligence mechanisms for accurate forecasting and trend analysis. However, researchers and data scientists face challenges in finding appropriate EO datasets due to inconsistent metadata structures and varied keyword descriptions. This misalignment hinders the discoverability and usability of EO data. To address this challenge, the EIFFEL ontology (EIFF-O) is proposed. EIFF-O introduces taxonomies and ontologies to provide (i) global classification of EO data and (ii) linkage between different datasets through common concepts. The taxonomies specified by the European Association of Remote Sensing Companies (EARSC) have been formalized and implemented in EIFF-O. Additionally, EIFF-O incorporates:1.An Essential Climate Variable (ECV) ontology, defined by the Global Climate Observing System (GCOS), is embedded and tailored for Climate Change (CC) applications.2.The Sustainable Development Goals (SDG) ontology is included to facilitate linking datasets to specific targets.3.The ontology extends schema.org vocabularies and promotes the use of JavaScript Object Notation for Linked Data (JSON-LD) formats for semantic web integration. EIFF-O provides a unified framework that enhances the discoverability, usability, and application of EO datasets. The implementation of EIFF-O allows data providers and users to bridge the gap between varied metadata descriptions and structured classification, thereby facilitating better linkage and integration of EO datasets. The EIFFEL ontology represents a significant advancement in the organization and application of EO datasets. By embedding ECV and SDG ontologies and leveraging semantic web technologies, EIFF-O not only streamlines the data discovery process but also supports diverse applications, particularly in Climate Change monitoring and Sustainable Development Goals achievement. The open-source nature of the ontology and its associated tools promotes rapid adoption among developers.

Enriching Earth observation datasets through semantics for climate change applications: The EIFFEL ontology

Background Earth Observation (EO) datasets have become vital for decision support applications, particularly from open satellite portals that provide extensive historical datasets. These datasets can be integrated with in-situ data to power artificial intelligence mechanisms for accurate forecasting and trend analysis. However, researchers and data scientists face challenges in finding appropriate EO datasets due to inconsistent metadata structures and varied keyword descriptions. This misalignment hinders the discoverability and usability of EO data. Methods To address this challenge, the EIFFEL ontology (EIFF-O) is proposed. EIFF-O introduces taxonomies and ontologies to provide (i) global classification of EO data and (ii) linkage between different datasets through common concepts. The taxonomies specified by the European Association of Remote Sensing Companies (EARSC) have been formalized and implemented in EIFF-O. Additionally, EIFF-O incorporates: 1. An Essential Climate Variable (ECV) ontology, defined by the Global Climate Observing System (GCOS), is embedded and tailored for Climate Change (CC) applications. 2. The Sustainable Development Goals (SDG) ontology is included to facilitate linking datasets to specific targets. 3. The ontology extends schema.org vocabularies and promotes the use of JavaScript Object Notation for Linked Data (JSON-LD) formats for semantic web integration. Results EIFF-O provides a unified framework that enhances the discoverability, usability, and application of EO datasets. The implementation of EIFF-O allows data providers and users to bridge the gap between varied metadata descriptions and structured classification, thereby facilitating better linkage and integration of EO datasets. Conclusions The EIFFEL ontology represents a significant advancement in the organization and application of EO datasets. By embedding ECV and SDG ontologies and leveraging semantic web technologies, EIFF-O not only streamlines the data discovery process but also supports diverse applications, particularly in Climate Change monitoring and Sustainable Development Goals achievement. The open-source nature of the ontology and its associated tools promotes rapid adoption among developers

Cross Modal Data Discovery over Structured and Unstructured Data Lakes

Organizations are collecting increasingly large amounts of data for data-driven decision making. These data are often dumped into a centralized repository, e.g., a data lake, consisting of thousands of structured and unstructured datasets. Perversely, such mixture makes the problem of discovering tables or documents that are relevant to a user's query very challenging. Despite the recent efforts in data discovery , the problem remains widely open especially in the two fronts of (1) discovering relationships and relatedness across structured and unstructured datasets-where existing techniques suffer from either scalability, being customized for a specific problem type (e.g., entity matching or data integration), or demolishing the structural properties on its way, and (2) developing a holistic system for integrating various similarity measurements and sketches in an effective way to boost the discovery accuracy. In this paper, we propose a new data discovery system, named CMDL, for addressing these two limitations. CMDL supports the data discovery process over both structured and unstructured data while retaining the structural properties of tables. As a result, CMDL is the only system to date that empowers end-users to seamlessly pipeline the discovery tasks across the two modalities. We propose a novel multi-modal embedding representation that captures the similarities between text documents and tabular columns. The model training relies on labeled datasets generated though weak supervision , and thus the system is domain agnostic and easily generalizable. We evaluate CMDL on three real-world data lakes with diverse applications and show that our system is significantly more effective for cross-modality discovery compared to the search-based baseline techniques. Moreover, CMDL is more accurate and robust to different data types and distributions compared to the state-of-the-art systems that are limited to only the structured datasets.

Survey of Process of Data Discovery and Environmental Decision Support Systems

The process of data discovery is an approach to extracting knowledge, valid, and usable information from large amounts of data, using automatic or semi-automatic methods. This article is an inventory of the different information extraction processes encountered in the literature for different fields of application and for the development of environmental informatics. Following an analysis between the different models, we can summarize the existing models with a proposal for a process that exploits the strengths of the different processes.

FSS-SDD: fuzzy-based semantic search for secure data discovery from outsourced cloud data

In current decade, cloud computing has become more prevalent and utilized for managing more sensitive data that are stored over cloud storage. While outsourcing the sensitive and private data on to the cloud, security is the major factor to be concentrated in efficient data search using keywords. As is auspicious, security of selective keyword search can be achieved using the powerful cryptographic technique called encryption. That is, the data that are to be outsourced have to be encrypted before storing it on cloud. Symmetric encryption is used between the data owner and the customer, whereas asymmetric encryption is used between the data owner and the cloud server. In such cases, the data discovery and retrieval have become more stimulating process. However, there are many methodologies developed for searching the encrypted outsourced data from the cloud. In this paper, the efficient data discovery process is focused and enhanced with the developed model called fuzzy-based semantic search for secure data discovery (FSS-SDD). A multi-valued logic called fuzzy logic is used, which has truth values of variables ranging between 0 and 1. It is used where the truth values range between completely true and completely false. The fuzzy logic-based semantic search improves the searching experience of the end user, by finding and retrieving the exact matching files for corresponding search files given by the user. Moreover, the model discovers the closely relevant matches using the semantic similarities, in cases, when the exact matches are not avail. For reducing the false positive rates, grading mechanism is enforced. Using the proposed FSS-SDD model, the processing overhead on new updates is effectively reduced and security in data retrieval is guaranteed.

A Cloud-Based Framework for Large-Scale Log Mining through Apache Spark and Elasticsearch

The volume, variety, and velocity of different data, e.g., simulation data, observation data, and social media data, are growing ever faster, posing grand challenges for data discovery. An increasing trend in data discovery is to mine hidden relationships among users and metadata from the web usage logs to support the data discovery process. Web usage log mining is the process of reconstructing sessions from raw logs and finding interesting patterns or implicit linkages. The mining results play an important role in improving quality of search-related components, e.g., ranking, query suggestion, and recommendation. While researches were done in the data discovery domain, collecting and analyzing logs efficiently remains a challenge because (1) the volume of web usage logs continues to grow as long as users access the data; (2) the dynamic volume of logs requires on-demand computing resources for mining tasks; (3) the mining process is compute-intensive and time-intensive. To speed up the mining process, we propose a cloud-based log-mining framework using Apache Spark and Elasticsearch. In addition, a data partition paradigm, logPartitioner, is designed to solve the data imbalance problem in data parallelism. As a proof of concept, oceanographic data search and access logs are chosen to validate performance of the proposed parallel log-mining framework.

Exploring Cancer Risk Factors using Data Mining Techniques: A Case Study from Yemen

Data Mining is a hot scientific research topic that has many applications in various life aspects. Healthcare and medicine are among those aspects that attracted data mining researchers who sought to solve decision-making problems. Cancer diagnosis, treatment, and prediction are procedures that have been using data mining for decades. In Yemen, some cancer risk factors seem to be different from those in other parts of the world. By mining the data available at National Cancer Control Foundation (NCCF), useful knowledge have been extracted. In this paper, decision tree classification was selected for building a model to predict cancer risk factors. As the NCCF database contained data describe some social life aspects, environmental circumstances, lifestyle, etc., mining those data can contribute in the endeavors of clearing ambiguity about cancer risk factors in Yemen. The informative attributes that were selected for model building included gender, marital status, number of family members, province, chewing Qat, chewing tobacco (Shamaa), smoking, age, relatives with cancer, and cancer class. These data was prepared for Knowledge Data Discovery process. Then, it was prepared for feeding into C4.5 learning algorithms. The results shown that smoking, chewing tobacco (Shamaa), province of residence, marital status, and age are the most important cancer risk factors. The model produced found of high performance. In addition, the rules extracted from the model tree can also be of high value for both people and healthcare sector.

Exploring Cancer Risk Factors using Data Mining Techniques: A Case Study from Yemen

Data Mining is a hot scientific research topic that has many applications in various life aspects. Healthcare and medicine are among those aspects that attracted data mining researchers who sought to solve decision-making problems. Cancer diagnosis, treatment, and prediction are procedures that have been using data mining for decades. In Yemen, some cancer risk factors seem to be different from those in other parts of the world. By mining the data available at National Cancer Control Foundation (NCCF), useful knowledge have been extracted. In this paper, decision tree classification was selected for building a model to predict cancer risk factors. As the NCCF database contained data describe some social life aspects, environmental circumstances, lifestyle, etc., mining those data can contribute in the endeavors of clearing ambiguity about cancer risk factors in Yemen. The informative attributes that were selected for model building included gender, marital status, number of family members, province, chewing Qat, chewing tobacco (Shamaa), smoking, age, relatives with cancer, and cancer class. These data was prepared for Knowledge Data Discovery process. Then, it was prepared for feeding into C4.5 learning algorithms. The results shown that smoking, chewing tobacco (Shamaa), province of residence, marital status, and age are the most important cancer risk factors. The model produced found of high performance. In addition, the rules extracted from the model tree can also be of high value for both people and healthcare sector.

Performance of IF-Postdiffset and R-Eclat Variants in Large Dataset

Pattern mining refers to a subfield of data mining that uncovers interesting, unexpected, and useful patterns from transaction databases. Such patterns reflect frequent and infrequent patterns. An abundant literature has dedicated in frequent pattern mining and tremendous efficient algorithms for frequent itemset mining in the transaction database. Nonetheless, the infrequent pattern mining has emerged to be an interesting issue in discovering patterns that rarely occur in the transaction database. More researchers reckon that rare pattern occurrences may offer valuable information in knowledge data discovery process. The R-Eclat is a novel algorithm that determines infrequent patterns in the transaction database. The multiple variants in the R-Eclat algorithm generate varied performances in infrequent mining patterns. This paper proposes IF-Postdiffset as a new variant in R-Eclat algorithm. This paper also highlights the performance of infrequent mining pattern from the transaction database among different variants of the R-Eclat algorithm regarding its execution time.

Improved group-based cooperative caching scheme for mobile ad hoc networks

Data caching is a popular technique that improves data accessibility in wired or wireless networks. However, in mobile ad hoc networks, improvement in access latency and cache hit ratio may diminish because of the mobility and limited cache space of mobile hosts (MHs). In this paper, an improved cooperative caching scheme called group-based cooperative caching (GCC) is proposed to generalize and enhance the performance of most group-based caching schemes. GCC allows MHs and their neighbors to form a group, and exchange a bitmap data directory periodically used for proposed algorithms, such as the process of data discovery, and cache placement and replacement. The goal is to reduce the access latency of data requests and efficiently use available caching space among MH groups. Two optimization techniques are also developed for GCC to reduce computation and communication overheads. The first technique compresses the directories using an aggregate bitmap. The second employs multi-point relays to develop a forwarding node selection scheme to reduce the number of broadcast messages inside the group. Our simulation results show that the optimized GCC yields better results than existing cooperative caching schemes in terms of cache hit ratio, access latency, and average hop count.

Aim4GDI: Facilitating the Synthesis of GDI Resources through Mapping and Superimpositions of Metadata Summaries

In the world of geospatial data infrastructures, geoportals are developed to facilitate access and use of geospatial resources, including data. The design and implementation of effective and efficient geoportals are becoming crucial. Providing possibilities for users to organize and integrate available resources can arguably enhance the process of data discovery and spatial analysis required, and hence, is the focus of this paper. We argue that the synthesis of summaries of distributed datasets, map and feature services through the geoportal can present a coherent view to support data discovery. For this purpose, the atlas metaphor is used as an indexing server and integration interface of summaries. Thematic maps and a storyteller view are accessible through the atlas metaphor. They can be used to provide a supporting context for data discovery and access purposes.

Looking below the Moon's surface with radar

Giovanni, the Goddard Earth Sciences Data and Information Services Center (GES DISC) Interactive Online Visualization and Analysis Infrastructure, has provided researchers with advanced capabilities to perform data exploration and analysis with observational data from NASA Earth observation satellites. In the past 5–10 years, examining geophysical events and processes with remote‐sensing data required a multistep process of data discovery, data acquisition, data management, and ultimately data analysis. Giovanni accelerates this process by enabling basic visualization and analysis directly on the World Wide Web. In the last two years, Giovanni has added new data acquisition functions and expanded analysis options to increase its usefulness to the Earth science research community.

Online analysis enhances use of NASA Earth science data

Giovanni, the Goddard Earth Sciences Data and Information Services Center (GES DISC) Interactive Online Visualization and Analysis Infrastructure, has provided researchers with advanced capabilities to perform data exploration and analysis with observational data from NASA Earth observation satellites. In the past 5–10 years, examining geophysical events and processes with remote‐sensing data required a multistep process of data discovery, data acquisition, data management, and ultimately data analysis. Giovanni accelerates this process by enabling basic visualization and analysis directly on the World Wide Web. In the last two years, Giovanni has added new data acquisition functions and expanded analysis options to increase its usefulness to the Earth science research community.

Distributed Spatial Catalog Service on the CORBA Object Bus

Spatial catalog service is a necessary component for distributed geographic information. Spatial catalog services provide the capabilities for organizing and managing spatial metadata. Catalog services simplify the process of data discovery in a large spatial database, facilitate data transfer, and help data fusion. This paper presents a software architecture for spatial catalog services. We first discuss the general concept of spatial catalog facilities and the logical model of spatial catalog services. We then introduce the distributed object-oriented model for spatial catalog services based on the industrial standard Common Object Request Broker Architecture (CORBA). The core components of spatial catalog services, catalog and catalog registry, are described in detail with the CORBA Interface Definition Language (IDL). Examples are given to illustrate the structure of the catalog component and its relations with other service components in a distributed GIS. The model provides a common interface to spatial catalogs and related data no matter how catalogs are defined, what formats they use, and where they are located. It provides crucial services in a distributed Geographic Information System.