Metadata Catalog Research Articles

Evaluating non-relational storage technology for HEP metadata and meta-data catalog

Large-scale scientific experiments produce vast volumes of data. These data are stored, processed and analyzed in a distributed computing environment. The life cycle of experiment is managed by specialized software like Distributed Data Management and Workload Management Systems. In order to be interpreted and mined, experimental data must be accompanied by auxiliary metadata, which are recorded at each data processing step. Metadata describes scientific data and represent scientific objects or results of scientific experiments, allowing them to be shared by various applications, to be recorded in databases or published via Web. Processing and analysis of constantly growing volume of auxiliary metadata is a challenging task, not simpler than the management and processing of experimental data itself. Furthermore, metadata sources are often loosely coupled and potentially may lead to an end-user inconsistency in combined information queries. To aggregate and synthesize a range of primary metadata sources, and enhance them with flexible schema-less addition of aggregated data, we are developing the Data Knowledge Base architecture serving as the intelligence behind GUIs and APIs.

Achievements and challenges in the integration, reuse and synthesis of vegetation plot data

AbstractAimsI aim to review vegetation plot data discovery, the major international efforts to integrate these data, some of the remaining barriers to data integration, reuse and synthesis and how they can be overcome, and some of the emergent issues associated with data attribution and acknowledgement for data providers, users and aggregators.ResultsVegetation plot data from 231 databases containing over three million plot records can be discovered via the metadata catalogue of the Global Index of Vegetation‐Plot Databases (GIVD). Major efforts to integrate data at national and international scales are well underway, including the North American VegBank, the European Vegetation Archive, the Botanical Information and Ecology Network (BIEN) and sPlot. Barriers to data reuse and synthesis remain: the most important are missing or incorrect geographic coordinates (geo‐coordinates) and inconsistencies in plant names. Many scientific journals now require the data underpinning published results to be archived in a publically accessible location via a digital object identifier (DOI). Such policies may be at odds with those of vegetation plot databases and funding agencies. The linkage between the New Zealand National Vegetation Survey Databank and an institutional data repository illustrates one solution to satisfying journal requirements to make data publically available, while retaining a direct linkage to the source data archive.ConclusionsAlthough further progress needs to be made in digitising, publishing and integrating vegetation plot data, many once insurmountable barriers are rapidly being overcome. Developing effective solutions to the problems posed by changing taxonomic concepts in space and time is likely the most urgent requirement. Although changing journal requirements may result in vegetation plot data being archived in some form for a specific publication, this does not provide the integration required to enable data reuse and synthesis. For vegetation scientists, a recommended best practice is to archive plot data in an established vegetation plot repository as a first step, and when required, provide versioned data or summaries to meet journal requirements in a suitable repository with a clear linkage to the vegetation plot repository. The concepts outlined in this paper have wide‐ranging implications for other types of ecological data.

ENHANCED DATA DISCOVERABILITY FOR IN SITU HYPERSPECTRAL DATASETS

Field spectroscopic metadata is a central component in the quality assurance, reliability, and discoverability of hyperspectral data and the products derived from it. Cataloguing, mining, and interoperability of these datasets rely upon the robustness of metadata protocols for field spectroscopy, and on the software architecture to support the exchange of these datasets. Currently no standard for in situ spectroscopy data or metadata protocols exist. This inhibits the effective sharing of growing volumes of in situ spectroscopy datasets, to exploit the benefits of integrating with the evolving range of data sharing platforms. A core metadataset for field spectroscopy was introduced by Rasaiah et al., (2011-2015) with extended support for specific applications. This paper presents a prototype model for an OGC and ISO compliant platform-independent metadata discovery service aligned to the specific requirements of field spectroscopy. In this study, a proof-of-concept metadata catalogue has been described and deployed in a cloud-based architecture as a demonstration of an operationalized field spectroscopy metadata standard and web-based discovery service.

ENHANCED DATA DISCOVERABILITY FOR IN SITU HYPERSPECTRAL DATASETS

Abstract. Field spectroscopic metadata is a central component in the quality assurance, reliability, and discoverability of hyperspectral data and the products derived from it. Cataloguing, mining, and interoperability of these datasets rely upon the robustness of metadata protocols for field spectroscopy, and on the software architecture to support the exchange of these datasets. Currently no standard for in situ spectroscopy data or metadata protocols exist. This inhibits the effective sharing of growing volumes of in situ spectroscopy datasets, to exploit the benefits of integrating with the evolving range of data sharing platforms. A core metadataset for field spectroscopy was introduced by Rasaiah et al., (2011-2015) with extended support for specific applications. This paper presents a prototype model for an OGC and ISO compliant platform-independent metadata discovery service aligned to the specific requirements of field spectroscopy. In this study, a proof-of-concept metadata catalogue has been described and deployed in a cloud-based architecture as a demonstration of an operationalized field spectroscopy metadata standard and web-based discovery service.

Directory search performance optimization of AMGA for the Belle II experiment

AMGA (ARDA Metadata Grid Application) is a grid metadata catalogue system that has been developed as a component of the EU FP7 EMI consortium based on the requirements of the HEP (High-Energy Physics) and the biomedical user communities. Currently, AMGA is exploited to manage the metadata in the gBasf2 framework at the Belle II experiment, one of the largest particle physics experiments in the world. In this paper, we present our efforts to optimize the metadata query performance of AMGA to better support the massive MC Campaign of the Belle II experiment. Although AMGA exhibits very outstanding performance for a relatively small amount of data, as the number of directories and the metadata size increase (e.g. hundreds of thousands of directories) during the MC Campaign, AMGA suffers from severe query processing performance degradation. To address this problem, we modified the query search mechanism and the database scheme of AMGA to provide dramatic improvements of metadata search performance and query response time. Throughout our comparative performance analysis of metadata search operations, we show that AMGA can be an optimal solution for a metadata catalogue in a large-scale scientific experimental framework

Belle II production system

The Belle II experiment will record a similar quantity of data to LHC experiments and will acquire it at similar rates. This requires considerable computing, storage and network resources to handle not only data created by the experiment but also considerable amounts of simulated data. Consequently Belle II employs a distributed computing system to provide the resources coordinated by the the DIRAC interware. DIRAC is a general software framework that provides a unified interface among heterogeneous computing resources. In addition to the well proven DIRAC software stack, Belle II is developing its own extension called BelleDIRAC. BelleDIRAC provides a transparent user experience for the Belle II analysis framework (basf2) on various environments and gives access to file information managed by LFC and AMGA metadata catalog. By unifying DIRAC and BelleDIRAC functionalities, Belle II plans to operate an automated mass data processing framework named a “production system”. The Belle II production system enables large-scale raw data transfer from experimental site to raw data centers, followed by massive data processing, and smart data delivery to each remote site. The production system is also utilized for simulated data production and data analysis. Although development of the production system is still on-going, recently Belle II has prepared prototype version and evaluated it with a large scale simulated data production. In this presentation we will report the evaluation of the prototype system and future development plans.

Facilitating the production of ISO-compliant metadata of geospatial datasets

Metadata are recognized as an essential element to enable efficient and effective discovery of geospatial data published in spatial data infrastructures (SDI). However, metadata production is still perceived as a complex, tedious and time-consuming task. This typically results in little metadata production and can seriously hinder the objective of facilitating data discovery.In response to this issue, this paper presents a proof of concept based on an interoperable workflow between a data publication server and a metadata catalog to automatically generate ISO-compliant metadata.The proposed approach facilitates metadata creation by embedding this task in daily data management workflows; ensures that data and metadata are permanently up-to-date; significantly reduces the obstacles of metadata production; and potentially facilitates contributions to initiatives like the Global Earth Observation System of Systems (GEOSS) by making geospatial resources discoverable.

A Decentralized Framework for Semantic Web Services Discovery Using Mobile Agent

Web services discovery provided by the UDDI registries is relatively primitive. It does not take into account the continuous growth in the number of services on the Web. The UDDI standard has been proposed and used for Web service publication and discovery. However, it does not allow users to choose the best provider. It does not offer a mechanism to choose a Web service based on its quality. The standard also lacks of sufficient semantic description in the content of Web services, this lack makes it difficult to find and compose suitable Web services during analysis, search, and matching processes. In addition, a central UDDI suffers from one centralized point problem and the high cost of maintenance. To get around these problems, the authors propose in this paper a novel framework based on mobile agent and metadata catalogue for Web services discovery. Their approach is based on user profile in order to discover appropriate Web services, meeting customer requirements, in less time and taking into account the QoS properties.

Provenance metadata gathering and cataloguing of EFIT++ code execution

Journal publications, as the final product of research activity, are the result of an extensive complex modeling and data analysis effort. It is of paramount importance, therefore, to capture the origins and derivation of the published data in order to achieve high levels of scientific reproducibility, transparency, internal and external data reuse and dissemination. The consequence of the modern research paradigm is that high performance computing and data management systems, together with metadata cataloguing, have become crucial elements within the nuclear fusion scientific data lifecycle. This paper describes an approach to the task of automatically gathering and cataloguing provenance metadata, currently under development and testing at Culham Center for Fusion Energy. The approach is being applied to a machine-agnostic code that calculates the axisymmetric equilibrium force balance in tokamaks, EFIT++, as a proof of principle test. The proposed approach avoids any code instrumentation or modification. It is based on the observation and monitoring of input preparation, workflow and code execution, system calls, log file data collection and interaction with the version control system. Pre-processing, post-processing, and data export and storage are monitored during the code runtime. Input data signals are captured using a data distribution platform called IDAM. The final objective of the catalogue is to create a complete description of the modeling activity, including user comments, and the relationship between data output, the main experimental database and the execution environment. For an intershot or post-pulse analysis (∼1000 time slices, 65×65 grid, mpi execution n=8 cores) of a typical MAST pulse, the overhead in the code runtime caused by the Provenance Metadata Gathering System is less than 10%, the metadata/data size ratio is about ∼20%, which we consider to be reasonable according to the present literature. A visualization interface based on Gephi for catalogue interrogation, will be presented.

A FRAMEWORK FOR COLLABORATIVE REVIEW OF CANDIDATE EVENTS IN HIGH DATA RATE STREAMS: THE V-FASTR EXPERIMENT AS A CASE STUDY

ABSTRACT “Fast radio transients” are defined here as bright millisecond pulses of radio-frequency energy. These short-duration pulses can be produced by known objects such as pulsars or potentially by more exotic objects such as evaporating black holes. The identification and verification of such an event would be of great scientific value. This is one major goal of the Very Long Baseline Array (VLBA) Fast Transient Experiment (V-FASTR), a software-based detection system installed at the VLBA. V-FASTR uses a “commensal” (piggy-back) approach, analyzing all array data continually during routine VLBA observations and identifying candidate fast transient events. Raw data can be stored from a buffer memory, which enables a comprehensive off-line analysis. This is invaluable for validating the astrophysical origin of any detection. Candidates discovered by the automatic system must be reviewed each day by analysts to identify any promising signals that warrant a more in-depth investigation. To support the timely analysis of fast transient detection candidates by V-FASTR scientists, we have developed a metadata-driven, collaborative candidate review framework. The framework consists of a software pipeline for metadata processing composed of both open source software components and project-specific code written expressly to extract and catalog metadata from the incoming V-FASTR data products, and a web-based data portal that facilitates browsing and inspection of the available metadata for candidate events extracted from the VLBA radio data.

Intelligent polar cyberinfrastructure: enabling semantic search in geospatial metadata catalogue to support polar data discovery

Polar regions have garnered substantial research attention in recent years because they are key drivers of the Earth’s climate, a source of rich mineral resources, and the home of a variety of marine life. Nevertheless, global warming over the past century is pushing the polar systems towards a tipping point: the systems are at high-risk from melting snow and sea ice covers, permafrost thawing, and acidification of the Arctic oceans. To increase understanding of the polar environment, the National Science Foundation established a Polar Cyberinfrastructure (CI) program, aimed at utilizing advanced software architecture to support polar data analysis and decision-making. At the center of this Polar CI research are data resources and data discovery components that facilitate the search and retrieval of polar data. This paper reports our development of a semantic search tool that supports the intelligent discovery of polar datasets. This tool is built on latent semantic analysis techniques, which improves search performance by identifying hidden semantic associations between terminologies used in the various datasets’ metadata. The software tool is implemented using an object-oriented design pattern and has been successfully integrated into a popular open source metadata catalog as a new semantic search support. A semantic matrix is maintained persistently within the catalogue to store the semantic associations. A dynamic update mechanism was also developed to allow automated update of semantics once more metadata are loaded into or removed from the catalog. We explored the effects of rank reduction to the effectiveness of this semantic search module and demonstrated its better performance than the traditional search techniques.

Implementation of a webGIS service platform for high mountain climate research: the SHARE GeoNetwork project

The implementation of a webGIS service platform dedicated to the management and sharing of climatological data acquired by high elevation stations is the core of the Station at High Altitude for Research on the Environment (SHARE) GeoNetwork project, promoted by the Ev‐K2 CNR Committee. The web platform basically will provide three types of services: structured metadata archive, data and results from high‐altitude environments research and projects; access to high‐altitude Ev‐K2 CNR stations and creation of a network of existing stations; dedicated webGIS for geo‐referenced data collected during the research. High elevation environmental and territorial data and metadata are catalogued in a single integrated platform to get access to the information heritage of the SHARE project, using open‐source tools: Geonetwork for the metadata catalogue and webGIS resources, and the open‐source Weather and Water Database (WDB), developed by the Norwegian Meteorological Institute, for the database information system implementation. The information system is designed to have a main node, with the possibility to install relocated subsystems based on the same technology, named focal point of SHARE, which will contain metadata and data connected to the main node. In this study, a new structure of metadata for the description of the climatological stations is proposed and WDB adaptation and data preprocessing are described in detail, giving code and script samples.

Building a Disciplinary Metadata Standards Directory

The Research Data Alliance (RDA) Metadata Standards Directory Working Group (MSDWG) is building a directory of descriptive, discipline-specific metadata standards. The purpose of the directory is to promote the discovery, access and use of such standards, thereby improving the state of research data interoperability and reducing duplicative standards development work.This work builds upon the UK Digital Curation Centre's Disciplinary Metadata Catalogue, a resource created with much the same aim in mind. The first stage of the MSDWG's work was to update and extend the information contained in the catalogue. In the current, second stage, a new platform is being developed in order to extend the functionality of the directory beyond that of the catalogue, and to make it easier to maintain and sustain. Future work will include making the directory more amenable to use by automated tools.

The CMS Data Management System

The data management elements in CMS are scalable, modular, and designed to work together. The main components are PhEDEx, the data transfer and location system; the Data Booking Service (DBS), a metadata catalog; and the Data Aggregation Service (DAS), designed to aggregate views and provide them to users and services. Tens of thousands of samples have been cataloged and petabytes of data have been moved since the run began. The modular system has allowed the optimal use of appropriate underlying technologies. In this contribution we will discuss the use of both Oracle and NoSQL databases to implement the data management elements as well as the individual architectures chosen. We will discuss how the data management system functioned during the first run, and what improvements are planned in preparation for 2015.

Wide‐Field Plate Archive of the University Observatory Jena

AbstractWe present the archive of the wide‐field plate observations obtained at the University Observatory Jena, which is stored at the Astrophysical Institute of the Friedrich Schiller University Jena. The archive contains plates taken in the period February 1963 to December 1982 with the 60/90/180‐cm Schmidt telescope of the university observatory. A computer‐readable version of the plate metadata catalogue (for 1257 plates), the logbooks, as well as the digitized Schmidt plates in low and high resolution are now accessible to the astronomical community.This paper describes the properties of the archive, as well as the processing procedure of all plates in detail. (© 2014 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)

Design and implementation of AMGA metadata catalog system for interoperability in European middleware initiative

AMGA (ARDA Metadata Grid Application) is a grid metadata catalog system, that is part of the gLite middleware, developed for the EGEE (Enabling Grids for E-sciencE) project. Since the end of EGEE, gLite has participated in the European Middleware Initiative (EMI) with other major middleware systems such as ARC, dCache and UNICORE. Through AMGA’s three-year EMI collaboration, it has been actively developed, tested and deployed. Previously, AMGA had made progress by focusing on development for the purposes of function implementation and improvement as a metadata catalogue system. After participating in the EMI, it has designed and implemented middleware-interoperability functions according to EMI policies and strategies.

EPOS: A Novel Use of CERIF for Data-intensive Science

One of the key aspects of the approaching data-intensive science era is integration of data through interoperability of systems providing data products or visualization and processing services. Far from being simple, interoperability requires robust and scalable e-infrastructures capable of supporting it. In this work we present the case of EPOS, a plan for data integration in the field of Earth Sciences. We describe the design of its e-infrastructure and show its main characteristics. One of the main elements enabling the system to integrate data, data products and services is the metadata catalogue based on the CERIF metadata model. Such a model, modified to fit into the general e-infrastructure design, is part of a three-layer metadata architecture. CERIF guarantees a robust handling of metadata, which is in this case the key to the interoperability and to one of the feature of the EPOS system: the possibility of carrying on data intensive science orchestrating the distributed resources made available by EPOS data providers and stakeholders.

A Linked Data Approach for Geospatial Data Provenance

Geospatial data provenance records sources and processing steps that are used in deriving geospatial data products. In the Web of Data environment enabled by Linked Data technologies, sources and processing steps, such as geospatial data and geoprocessing services, can be published as part of the Web of Data. To take full advantages of the machine-understandable format and linkages among heterogeneous data items in the Web of Data, this paper proposes to publish geospatial data provenance into the Web of Data. In particular, it analyzes how a catalogue for provenance, i.e., geospatial data provenance managed by a geospatial metadata catalog service, can be published into the Web of Data using a Linked Data approach. Consequently, queries over linked geospatial data provenance are analyzed and tested to demonstrate the benefits of the approach.

An Interlinking Approach for Linked Geospatial Data

Abstract. Geospatial metadata from metadata catalogue can be published as part of Web of data used Linked Data technologies. The published data could be named as linked geospatial metadata. A key issue of Linked Data technologies is to create links among datasets. There are three important types of RDF links: relationship links, identity links, and vocabulary links. This paper proposes a matching method to construct linkages between linked geospatial metadata and geospatial datasets in the linking open data cloud (LOD). This matching method is based on semantic similarity to construct identity links. A matching algorithm using Tversky‘s contrast model and Jaro-Winkler distance is proposed and evaluated.

Digital Climate Atlas of the Carpathian Region

Abstract. The main goal of the CARPATCLIM (Climate of the Carpathian Region) project is to construct the gridded climatological database for the region in a daily temporal resolution for the period 1961–2010 by using 0.1° spatial resolution. The solution of this requirement as well as one of the final products of the CARPATCLIM project is a Digital Climate Atlas which is designed as the main entry point for all the gridded data and maps generated during the project, together with metadata for all data sets (original data as well as data created during the project). With respect to the INSPIRE (Infrastructure for Spatial Information in the European Community) directive, the Digital Climate Atlas is developed as a rich Web GIS (Geographic Information System) application based on modern Web standards offering all necessary tools for climate data visualization and extraction. Another important product of the CARPATCLIM project is the Metadata Catalog which is designed as a tool for searching of climate metadata by various parameters (i.e. period, variable, region etc.).