Provenance Metadata Research Articles

Associations Between the Principle of Provenance and Metadata: Practical Implementation and Cases

The principle of provenance is the fundamental theory of archives science, which also possesses important reference value in various areas of research and practice. This paper commences with investigating the core value of the principle of provenance, considering the principle to be the unity of theory and method and its core value lies in following historicalism and respecting essential attributes of the archives. Then the authors focus on the realm of government information management and explore the metadata implementation of the core value of the principle of provenance in the era of electronic records. In its essence, the principle of provenance requires full understanding and preservation of the source information of the record. In the age of electronic records this has ascended to the significance of obtaining information, which is often stabilized as metadata in electronic records, reflecting the process of the record’s creation, maintenance and long term preservation. Source information as the contextual information reflecting the record creation process, its typical form of expression in metadata is provenance metadata. Therefore the authors explore the implications of provenance metadata and its essential forms. The authors hold that source information is the indispensable characteristic information in records and archives, and also the essential form for the principle of provenance to exhibit the connotations of provenance. This paper also explains that provenance metadata has further realized the core value of the principle of provenance, playing important roles in ensuring the authenticity, integrity, reliability and usability of electronic records as well as being the basis of their long term preservation and usage. In addition, the authors make concrete exposition on the composition and function of provenance metadata, considering it to be composed of entities and elements, specifically including such metadata as institution, business, environment, relationship and time. Different provenance metadata entities and elements record the source information of the electronic record from its creation until transfer to archives, performing various important functions in government electronic record management. The authors further their research through analyzing and comparing the characteristics of provenance metadata in government electronic record metadata standards at international, national and local jurisdictions levels. Some feasible suggestions for future research are made in the conclusion.

Metadata management for scientific databases

Most scientific databases consist of datasets (or sources) which in turn include samples (or files) with an identical structure (or schema). In many cases, samples are associated with rich metadata, describing the process that leads to building them (e.g.: the experimental conditions used during sample generation). Metadata are typically used in scientific computations just for the initial data selection; at most, metadata about query results is recovered after executing the query, and associated with its results by post-processing. In this way, a large body of information that could be relevant for interpreting query results goes unused during query processing.In this paper, we present ScQL, a new algebraic relational language, whose operations apply to objects consisting of data–metadatapairs, by preserving such one-to-one correspondence throughout the computation. We formally define each operation and we describe an optimization, called meta-first, that may significantly reduce the query processing overhead by anticipating the use of metadata for selectively loading into the execution environment only those input samples that contribute to the result samples.In ScQL, metadata have the same relevance as data, and contribute to building query results; in this way, the resulting samples are systematically associated with metadata about either the specific input samples involved or about query processing, thereby yielding a new form of metadata provenance. We present many examples of use of ScQL, relative to several application domains, and we demonstrate the effectiveness of the meta-first optimization.

The R-based climate4R open framework for reproducible climate data access and post-processing

Climate-driven sectoral applications commonly require different types of climate data (e.g. observations, reanalysis, climate change projections) from different providers. Data access, harmonization and post-processing (e.g. bias correction) are time-consuming error-prone tasks requiring different specialized software tools at each stage of the data workflow, thus hindering reproducibility. Here we introduce climate4R, an R-based climate services oriented framework tailored to the needs of the vulnerability and impact assessment community that integrates in the same computing environment harmonized data access, post-processing, visualization and a provenance metadata model for traceability and reproducibility of results. climate4R allows accessing local and remote (OPeNDAP) data sources, such as the Santander User Data Gateway (UDG), a THREDDS-based service including a wide catalogue of popular datasets (e.g. ERA-Interim, CORDEX, etc.). This provides a unique comprehensive open framework for end-to-end sectoral reproducible applications. All the packages, data and documentation for reproducing the experiments in this paper are available from http://www.meteo.unican.es/climate4R.

Data linkage in medical science using the resource description framework: the AVERT model

There is an ongoing challenge as to how best manage and understand 'big data' in precision medicine settings. This paper describes the potential for a Linked Data approach, using a Resource Description Framework (RDF) model, to combine multiple datasets with temporal and spatial elements of varying dimensionality. This "AVERT model" provides a framework for converting multiple standalone files of various formats, from both clinicaland environmental settings, into a single data source. This data source can thereafter be queried effectively, shared with outside parties, more easily understood by multiple stakeholders using standardized vocabularies, incorporating provenance metadata and supporting temporo-spatial reasoning. The approach has further advantages in terms of data sharing, security and subsequent analysis. We use a case study relating to anti-Glomerular Basement Membrane (GBM) disease, a rare autoimmune condition, to illustrate a technical proof of concept for the AVERT model.

Entry Life-Cycle with automatic Change-History & Provenance Tracking in collaborative Semantic Web Content Management Systems as implemented in SOCCOMAS

SOCCOMAS is a ready-to-use Semantic Ontology-Controlled Content Management System (http://escience.biowikifarm.net/wiki/SOCCOMAS). Each web content management system (WCMS) run by SOCCOMAS is controlled by a set of ontologies and an accompanying Java-based middleware with the data housed in a Jena tuple store. The ontologies describe the behavior of the WCMS, including all of its input forms, input controls, data schemes and workflow processes (Fig. 1). Data is organized into different types of data entries, which represent collections of data referring to a particular material entity, for instance an individual specimen. SOCCOMAS implements a suite of general processes, which can be used to manage and organize all data entry types. One category of processes manages the life-cycle of a data entry, including all required for changing between the following possible entry states: current draft version; backup draft version; recycle bin draft version; deleted draft version; current published version; previously published version. current draft version; backup draft version; recycle bin draft version; deleted draft version; current published version; previously published version. The processes also allow a user to create a revised draft based on the current published version. Another category of processes automatically tracks the overall provenance (i.e. creator, authors, creation and publication date, contributers, relation between different versions, etc.) for each particular data entry. Additionally, on a significantly finer level of granularity, SOCCOMAS also tracks in a detailed change-history log all changes made to a particular data record at the level of individual input fields. All information (data, provenance metadata, change-history metadata) is stored based on Resource Description Framework (RDF) compliant data schemes into different named graphs (i.e. a URI under which triple statements are stored in the tuple store). All recorded information can be accessed through a SPARQL endpoint. All data entries are Linked Open Data and thus provide access to an HTML representation of the data for visualization in a web-browser or as a machine-readable RDF file. The ontology-controlled design of SOCCOMAS allows administrators to easily customize already existing templates for input forms of data entries, define new templates for new types of data entries, and define underlying RDF-compliant data schemes and apply them to each relevant input field. SOCCOMAS provides an engine for running and developing semantic WCMSs, where only ontology editing, but no middleware and front end programming, are required for adapting the WCMS to one's own specific requirements.

MetaStore: an adaptive metadata management framework for heterogeneous metadata models

In this paper, we present MetaStore, a metadata management framework for scientific data repositories. Scientific experiments are generating a deluge of data, and the handling of associated metadata is critical, as it enables discovering, analyzing, reusing, and sharing of scientific data. Moreover, metadata produced by scientific experiments are heterogeneous and subject to frequent changes, demanding a flexible data model. Existing metadata management systems provide a broad range of features for handling scientific metadata. However, the principal limitation of these systems is their architecture design that is restricted towards either a single or at the most a few standard metadata models. Support for handling different types of metadata models, i.e., administrative, descriptive, structural, and provenance metadata, and including community-specific metadata models is not possible with these systems. To address this challenge, we present MetaStore, an adaptive metadata management framework based on a NoSQL database and an RDF triple store. MetaStore provides a set of core functionalities to handle heterogeneous metadata models by automatically generating the necessary software code (services) and on-the-fly extends the functionality of the framework. To handle dynamic metadata and to control metadata quality, MetaStore also provides an extended set of functionalities such as enabling annotation of images and text by integrating the Web Annotation Data Model, allowing communities to define discipline-specific vocabularies using Simple Knowledge Organization System, and providing advanced search and analytical capabilities by integrating the ElasticSearch. To maximize the utilization of the data models supported by NoSQL databases, MetaStore automatically segregates the different categories of metadata in their corresponding data models. Complex provenance graphs and dynamic metadata are modeled and stored in an RDF triple store, whereas the static metadata is stored in a NoSQL database. For enabling large-scale harvesting (sharing) of metadata using the METS standard over the OAI-PMH protocol, MetaStore is designed OAI-compliant. Finally, to show the practical usability of the MetaStore framework and that the requirements from the research communities have been realized, we describe our experience in the adoption of MetaStore for three communities.

On the origin of annotations: A module-based approach to representing annotations in the Natural Language Processing Interchange Format (NIF)

Representing provenance information for data is of crucial importance for data reuse. This is in particular the case for language resources such as annotated corpora. NIF has been proposed as an RDF vocabulary to support the representation of text data together with annotations. However, NIF suffers from severe shortcomings with respect to its ability to represent provenance information. As a remedy to this, we present MOND, a new glue ontology that implements an interface between NIF and the PROV-O ontology to support the inclusion of provenance information into NIF annotated datasets. We first present an approach that reifies annotations and allows the attachment of any provenance metadata to annotations at arbitrary granularity. We show that this approach has an important drawback as it roughly doubles the size of the data. Building on this observation, we design the MOND glue ontology that implements a modular approach in which annotation metadata is not attached to single annotations but to modules that represent collections of annotations of the same type and origin. This yields a moderate increase in data size, while maintaining all the benefits of the first approach. We validate our approach on three use cases that represent prototypical needs in corpus work.

Enriching Agronomic Experiments with Data Provenance

Reproducibility is a major feature of Science. Even agronomic research of exemplary quality may have irreproducible empirical findings because of random or systematic error. The ability to reproduce agronomic experiments based on statistical data and legacy scripts are not easily achieved. We propose RFlow, a tool that aid researchers to manage, share, and enact the scientific experiments that encapsulate legacy R scripts. RFlow transparently captures provenance of scripts and endows experiments reproducibility. Unlike existing computational approaches, RFlow is non-intrusive, does not require users to change their working way, it wraps agronomic experiments in a scientific workflow system. Our computational experiments show that the tool can collect different types of provenance metadata of real experiments and enrich agronomic data with provenance metadata. This study shows the potential of RFlow to serve as the primary integration platform for legacy R scripts, with implications for other data- and compute-intensive agronomic projects.

A Belated Green Revolution for Cannabis: Virtual Genetic Resources to Fast-Track Cultivar Development.

Cannabis is a predominantly diecious phenotypically diverse domesticated genus with few if any extant natural populations. International narcotics conventions and associated legislation have constrained the establishment, characterization, and use of Cannabis genetic resource collections. This has resulted in the underutilization of genepool variability in cultivar development and has limited the inclusion of secondary genepools associated with genetic improvement strategies of the Green Revolution. The structured screening of ex situ germplasm and the exploitation of locally-adapted intraspecific traits is expected to facilitate the genetic improvement of Cannabis. However, limited attempts have been made to establish the full extent of genetic resources available for pre-breeding. We present a thorough critical review of Cannabis ex situ genetic resources, and discuss recommendations for conservation, pre-breeding characterization, and genetic analysis that will underpin future cultivar development. We consider East Asian germplasm to be a priority for conservation based on the prolonged historical cultivation of Cannabis in this region over a range of latitudes, along with the apparent high levels of genetic diversity and relatively low representation in published genetic resource collections. Seed cryopreservation could improve conservation by reducing hybridization and genetic drift that may occur during Cannabis germplasm regeneration. Given the unique legal status of Cannabis, we propose the establishment of a global virtual core collection based on the collation of consistent and comprehensive provenance meta-data and the adoption of high-throughput DNA sequencing technologies. This would enable representative core collections to be used for systematic phenotyping, and so underpin breeding strategies for the genetic improvement of Cannabis.

DisGeNET-RDF: harnessing the innovative power of the Semantic Web to explore the genetic basis of diseases.

Motivation: DisGeNET-RDF makes available knowledge on the genetic basis of human diseases in the Semantic Web. Gene-disease associations (GDAs) and their provenance metadata are published as human-readable and machine-processable web resources. The information on GDAs included in DisGeNET-RDF is interlinked to other biomedical databases to support the development of bioinformatics approaches for translational research through evidence-based exploitation of a rich and fully interconnected linked open data.Availability and implementation: http://rdf.disgenet.org/Contact: support@disgenet.org

An Ontology-Enabled Natural Language Processing Pipeline for Provenance Metadata Extraction from Biomedical Text (Short Paper).

Extraction of structured information from biomedical literature is a complex and challenging problem due to the complexity of biomedical domain and lack of appropriate natural language processing (NLP) techniques. High quality domain ontologies model both data and metadata information at a fine level of granularity, which can be effectively used to accurately extract structured information from biomedical text. Extraction of provenance metadata, which describes the history or source of information, from published articles is an important task to support scientific reproducibility. Reproducibility of results reported by previous research studies is a foundational component of scientific advancement. This is highlighted by the recent initiative by the US National Institutes of Health called "Principles of Rigor and Reproducibility". In this paper, we describe an effective approach to extract provenance metadata from published biomedical research literature using an ontology-enabled NLP platform as part of the Provenance for Clinical and Healthcare Research (ProvCaRe). The ProvCaRe-NLP tool extends the clinical Text Analysis and Knowledge Extraction System (cTAKES) platform using both provenance and biomedical domain ontologies. We demonstrate the effectiveness of ProvCaRe-NLP tool using a corpus of 20 peer-reviewed publications. The results of our evaluation demonstrate that the ProvCaRe-NLP tool has significantly higher recall in extracting provenance metadata as compared to existing NLP pipelines such as MetaMap.

Terra Populus: Workflows for Integrating and Harmonizing Geospatial Population and Environmental Data

The Terra Populus project (TerraPop) addresses a variety of data management, curation, and preservation challenges with respect to spatiotemporal population and environmental data. In this article, we describe our approaches to these challenges, with a particular focus on geospatial data workflows and associated provenance metadata. The goal of TerraPop is to enable research, learning, and policy analysis by providing integrated spatiotemporal data describing people and their environment. To do so, TerraPop is assembling a globe-spanning and temporally extensive collection of high-quality population and environmental data, ensuring good documentation, and developing a Web-based data access system that enables users to assemble customized integrated data sets drawing on a variety of data sources and formats. We describe TerraPop's collection strategies, detail the geospatial workflows involved in preparing data for ingest into the project database and those used to transform data across formats for dissemination, and discuss the system used to capture and manage provenance metadata throughout the project. A key aspect of the project is the development of global current and historical administrative unit boundaries that can be linked to census data. These boundaries serve as the linchpin of TerraPop's data integration strategy, and constitute an important data set in their own right.

Provenance metadata gathering and cataloguing of EFIT++ code execution

Journal publications, as the final product of research activity, are the result of an extensive complex modeling and data analysis effort. It is of paramount importance, therefore, to capture the origins and derivation of the published data in order to achieve high levels of scientific reproducibility, transparency, internal and external data reuse and dissemination. The consequence of the modern research paradigm is that high performance computing and data management systems, together with metadata cataloguing, have become crucial elements within the nuclear fusion scientific data lifecycle. This paper describes an approach to the task of automatically gathering and cataloguing provenance metadata, currently under development and testing at Culham Center for Fusion Energy. The approach is being applied to a machine-agnostic code that calculates the axisymmetric equilibrium force balance in tokamaks, EFIT++, as a proof of principle test. The proposed approach avoids any code instrumentation or modification. It is based on the observation and monitoring of input preparation, workflow and code execution, system calls, log file data collection and interaction with the version control system. Pre-processing, post-processing, and data export and storage are monitored during the code runtime. Input data signals are captured using a data distribution platform called IDAM. The final objective of the catalogue is to create a complete description of the modeling activity, including user comments, and the relationship between data output, the main experimental database and the execution environment. For an intershot or post-pulse analysis (∼1000 time slices, 65×65 grid, mpi execution n=8 cores) of a typical MAST pulse, the overhead in the code runtime caused by the Provenance Metadata Gathering System is less than 10%, the metadata/data size ratio is about ∼20%, which we consider to be reasonable according to the present literature. A visualization interface based on Gephi for catalogue interrogation, will be presented.

Modeling and Implementing Scientific Hypothesis

Computational Simulations are important tools that enable scientists to study complex phenomena about which few data is available or that require dangerous human interventions. They involve complex and heterogeneous components, including: mathematical equations, hypothesis, computational models and data. In order to support in-silico scientific research this complex environment needs to be modeled and have its data and metadata managed enabling model evolution, prediction analysis and decision-making. This paper proposes a scientific hypothesis conceptual model that allows scientists to represent the phenomenon been investigated, the hypotheses formulated in the attempt to explain it, and provides the ability to store results of experiment simulations with their corresponding provenance metadata. The proposed model supports scientific life-cycle through: provenance management, exchange of hypothesis as data, experiment reproducibility, model steering and simulation result analyses. A cardiovascular numerical simulation illustrates the applicability of the model and an initial implementation using SciDB is discussed.

Nanopublications for exposing experimental data in the life-sciences: a Huntington's Disease case study

Data from high throughput experiments often produce far more results than can ever appear in the main text or tables of a single research article. In these cases, the majority of new associations are often archived either as supplemental information in an arbitrary format or in publisher-independent databases that can be difficult to find. These data are not only lost from scientific discourse, but are also elusive to automated search, retrieval and processing. Here, we use the nanopublication model to make scientific assertions that were concluded from a workflow analysis of Huntington’s Disease data machine-readable, interoperable, and citable. We followed the nanopublication guidelines to semantically model our assertions as well as their provenance metadata and authorship. We demonstrate interoperability by linking nanopublication provenance to the Research Object model. These results indicate that nanopublications can provide an incentive for researchers to expose data that is interoperable and machine-readable for future use and preservation for which they can get credits for their effort. Nanopublications can have a leading role into hypotheses generation offering opportunities to produce large-scale data integration.Electronic supplementary materialThe online version of this article (doi:10.1186/2041-1480-6-5) contains supplementary material, which is available to authorized users.

Scientific provenance metadata capture and management using Semantic Web

Providing historical scientific information to deal with knowledge loss about scientific experiment is a strategical question in scientific research. One key aspect is the reuse of generated knowledge. Provenance data, extracted from multiple sources, must be reused in order to provide strategical information for scientists, including also inferred information based on these data. SciProv consists of an architecture that aims to interact with Scientific Workflow Management Systems SWfMS for capturing and managing provenance metadata. This work adopts an approach based on a standard model to represent the lineage and to handle distributed and heterogeneous provenance metadata. Using ontologies and inference engines, SciProv provides resources to make inferences about provenance data and to obtain important results considering the extraction of information beyond those that are explicitly registered. The paper also presents proof of concept in bioinformatics domain.

Towards the unification of sequence-based classification and sequence-based identification of host-associated microorganisms.

Towards the unification of sequence-based classification and sequence-based identification of host-associated microorganisms.

Implementation of Geospatial Data Provenance in a Web Service Workflow Environment With ISO 19115 and ISO 19115-2 Lineage Model

Data provenance, also called data lineage, records the derivation history of a data product. In the earth science domain, geospatial data provenance is important because it plays a significant role in data quality and usability evaluation, data trail audition, workflow replication, and product reproducibility. The generation of the geospatial provenance metadata is usually coupled with the execution of geo-processing workflow. Their symbiotic relationship makes them complementary to each other and promises great benefit once they are integrated. However, the heterogeneity of data and computing resources in the distributed environment constructed under the service-oriented architecture (SOA) brings a great challenge to resource integration. Specifically, the issues, such as the lack of interoperability and compatibility among provenance metadata models and between provenance and workflow, create obstacles for the integration of provenance, and geo-processing workflow. In order to tackle these issues, on one hand, this paper breaks the provenance heterogeneity through recording provenance information in a standard lineage model defined in ISO 19115:2003 and ISO 19115-2:2009 standards. On the other hand, this paper bridges the gap between provenance and geo-processing workflow through extending both workflow language and service interface, making it possible for the automatic capture of provenance information in the geospatial web service environment. The proposed method is implemented in the GeoBrain, a SOA-based geospatial web service system. The testing result from implementation shows that the geospatial provenance information is successfully captured throughout the life cycle of geo-processing workflows and properly recorded in the ISO standard lineage model.

Architecture design of a user-orientated electronic laboratory notebook: A case study within an atmospheric chemistry community

Data is particularly valuable to scientists when details of its provenance are known. This research concerned deploying a user-orientated electronic laboratory notebook (ELN) system within a scientific community. The ELN system supported the capture and retrieval of semantic metadata describing the provenance of the modelling activities of scientists within that community. The research was grounded within the atmospheric chemistry community but has applicability to other communities using an iterative model development process. The ELN system involved the automatic capture of metadata concerning the modelling process together with inline annotations added by the modeller explaining the reasoning for modelling decisions at each step of the process. A full realisation of the ELN system was built and evaluated by members of the atmospheric chemistry community. In order to promote reusability the ELN system architecture had domain-independent as well as domain-dependent elements. An ontology (in OWL) was used to ensure that the specific terminology of the community was used within the provenance metadata and also that it was used consistently. Other domain-independent elements of the architecture included a dynamic graphic interface that allowed the modeller to view his/her modelling history. This was recorded as a set of nodes each pointing to the stored provenance metadata associated with a specific simulation run. In addition, there was an innovative mechanism that enabled the modeller to navigate through the various nodes. The navigation process supported making comparisons between different nodes: a facility that users found particularly valuable. Members of the atmospheric chemistry community took part in a two-day summative evaluation of the ELN system. This confirmed its value to the modellers and it is now being introduced more widely across the modelling community. In addition, the research proposes a methodology for transferring this ELN system to other modelling communities making use of the domain-independent elements of the architecture.

Environmental Informatics

Environmental informatics uses large multidimensional, complex datasets to study environmental problems, which can be both discrete and continuous in space or time. These datasets and their requisite metadata can be managed by queryable databases. Geospatial Web application programming interfaces (APIs) provide remote access to dynamic subsets of environmental information. Persistent identifiers make data citable. The storage-computing trade-off is now heavily skewed in favor of moving calculations to the data. Provenance metadata help determine a data object's reliability and trustworthiness. Rising atmospheric CO2, the Antarctic ozone hole, and Gulf Stream warm-core rings were all discovered by analyzing long-term datasets. Similar work continues on mapping evapotranspiration and snow water equivalent. In these “fourth paradigm” problems, data (especially data collected operationally) drive hypothesis formation. Making data available requires new discovery mechanisms and policies favoring data sharing. Cloud computing and array-friendly databases will help bring processing to the data. Ubiquitous location sensing and geotagging will help turn citizen scientists into environmental information collectors.