Metadata Quality Research Articles

Metadata remediation through migration, post-migration or necessary clean-up: A roadmap for success

Metadata clean-up (or remediation) projects can be time-consuming, stressful and, at some point, inevitable if metadata are retained. Having a plan before starting is paramount not only to a successful project, but also to prevent the waste of valuable resources. As this paper will show, determining which aspects of a clean-up project are most important is the key to ending the project with quality metadata — regardless of the metadata schema or metadata management system or the rationale for the clean-up.

FAIRness in digital forensics datasets’ metadata – and how to improve it

The availability of research data (datasets) and compliance with FAIR principles—Findability, Accessibility, Interoperability, and Reusability—is critical to progressing digital forensics. This study evaluates metadata completeness and assesses the alignment with the FAIR principles using all 212 datasets from NIST's Computer Forensic Reference DataSet Portal (CFReDS). The findings underscore deficiencies in metadata quality and FAIR compliance, emphasizing the need for improved data management standards. Based on our critical review, we then propose and discuss various approaches to improve the status quo.

Completeness degree of publication metadata in eight free-access scholarly databases

Abstract The main objective of this study is to compare the amount of metadata and the completeness degree of research publications in new academic databases. Using a quantitative approach, we selected a random Crossref sample of more than 115,000 records, which was then searched in seven databases (Dimensions, Google Scholar, Microsoft Academic, OpenAlex, Scilit, Semantic Scholar, and The Lens). Seven characteristics were analyzed (abstract, access, bibliographic info, document type, publication date, language, and identifiers), to observe fields that describe this information, the completeness rate of these fields, and the agreement among databases. The results show that academic search engines (Google Scholar, Microsoft Academic, and Semantic Scholar) gather less information and have a low degree of completeness. Conversely, third-party databases (Dimensions, OpenAlex, Scilit, and The Lens) have more metadata quality and a higher completeness rate. We conclude that academic search engines lack the ability to retrieve reliable descriptive data by crawling the web, and the main problem of third-party databases is the loss of information derived from integrating different sources.

Denoising of Geochemical Data using Deep Learning–Implications for Regional Surveys

Regional geochemical surveys generate large amounts of data that can be used for a number of purposes such as to guide mineral exploration. Modern surveys are typically designed to permit quantification of data uncertainty through data quality metrics by using quality assurance and quality control (QA/QC) methods. However, these metrics, such as data accuracy and precision, are obtained through the data generation phase. Consequently, it is unclear how residual uncertainty in geochemical data can be minimized (denoised). This is a limitation to propagating uncertainty through downstream activities, particularly through complex models, which can result from the usage of artificial intelligence-based methods. This study aims to develop a deep learning-based method to examine and quantify uncertainty contained in geochemical survey data. Specifically, we demonstrate that: (1) autoencoders can reduce or modulate geochemical data uncertainty; (2) a reduction in uncertainty is observable in the spatial domain as a decrease of the nugget; and (3) a clear data reconstruction regime of the autoencoder can be identified that is strongly associated with data denoising, as opposed to the removal of useful events in data, such as meaningful geochemical anomalies. Our method to post-hoc denoising of geochemical data using deep learning is simple, clear and consistent, with the amount of denoising guided by highly interpretable metrics and existing frameworks of scientific data quality. Consequently, variably denoised data, as well as the original data, could be fed into a single downstream workflow (e.g., mapping, general data analysis or mineral prospectivity mapping), and the differences in the outcome can be subsequently quantified to propagate data uncertainty.

Data reduction in protein serial crystallography.

Serial crystallography (SX) has become an established technique for protein structure determination, especially when dealing with small or radiation-sensitive crystals and investigating fast or irreversible protein dynamics. The advent of newly developed multi-megapixel X-ray area detectors, capable of capturing over 1000 images per second, has brought about substantial benefits. However, this advancement also entails a notable increase in the volume of collected data. Today, up to 2 PB of data per experiment could be easily obtained under efficient operating conditions. The combined costs associated with storing data from multiple experiments provide a compelling incentive to develop strategies that effectively reduce the amount of data stored on disk while maintaining the quality of scientific outcomes. Lossless data-compression methods are designed to preserve the information content of the data but often struggle to achieve a high compression ratio when applied to experimental data that contain noise. Conversely, lossy compression methods offer the potential to greatly reduce the data volume. Nonetheless, it is vital to thoroughly assess the impact of data quality and scientific outcomes when employing lossy compression, as it inherently involves discarding information. The evaluation of lossy compression effects on data requires proper data quality metrics. In our research, we assess various approaches for both lossless and lossy compression techniques applied to SX data, and equally importantly, we describe metrics suitable for evaluating SX data quality.

A Bayesian Approach to Estimate Maternal Mortality Globally Using National Civil Registration Vital Statistics Data Accounting for Reporting Errors

Estimation and monitoring of mortality, across multiple population-periods, is vital to improving national and global health outcomes. Reducing maternal mortality is a key part of the Sustainable Development Goals (SGDs), which commit countries and international agencies to monitor progress toward improvement of global maternal health outcomes. Country-specific estimates of maternal mortality are used for monitoring country progress. Civil registration vital statistics (CRVS) data provide valuable information on maternal mortality but are often subject to substantial reporting errors due to misclassification of maternal deaths or incompleteness of CRVS-reporting on deaths to women of reproductive age. Motivated by the challenge of estimating maternal mortality using error-prone CRVS data, this article introduces a Bayesian two-stage approach to obtain country-specific trends in maternal mortality ratios (MMRs), accounting for CRVS-related misclassification errors. In the first stage, we produce country-specific trends of CRVS-related data quality metrics, in terms sensitivity and specificity, through a Bayesian hierarchical misclassification model. In the second stage, we produce data-driven country trends in MMRs and proportion maternal deaths out of all-cause deaths, which are constructed using a Bayesian maternal mortality estimation model, accounting for CRVS-related data quality assessed in Stage 1. We present the approach to estimating maternal mortality and illustrate its use for several country case studies. This approach is used by the United National Maternal Mortality Interagency Group (UN-MMEIG) to produce maternal mortality estimates for global monitoring.

FAIR+R: Making Clinical Data Reliable Through Qualitative Metadata.

Metadata are often the first access to data repositories for researchers within secondary use. Through automatic metadata generation and metadata harvesting the amount of data about data has been growing ever since. In order to make data not only FAIR but also reliable, the aspect of metadata quality has to be considered. But as earlier assessments of metadata of different repositories showed, metadata quality still lacks behind its capability. Providing an extensive literature review the authors conclude nine measures to assess metadata in relation to clinical care repositories, such as Medical Data Integration Centers (MeDICs). Proceeding from these measures the authors propose an addition of the FAIR Guiding Principles by adding a fifth block for Reliability including three principles, that resulted from the measures presented. The results form the basis for the future work of an assessment of metadata, that is stored in a MeDIC.

Repairing raw metadata for metadata management

With the exponential growth of data production, the generation of metadata has become an integral part of the process. Metadata plays a crucial role in facilitating enhanced data analytics, data integration, and resource management by offering valuable insights. However, inconsistencies arise due to deviations from standards in metadata recording, including missing attribute information, publishing URLs, and provenance. Furthermore, the recorded metadata may exhibit inconsistencies, such as varied value formats, special characters, and inaccurately entered values. Addressing these inconsistencies through metadata preparation can greatly enhance the user experience during data management tasks.This paper introduces MDPrep, a system that explores the usability and applicability of data preparation techniques in improving metadata quality. Our approach involves three steps: (1) detecting and identifying problematic metadata elements and structural issues, (2) employing a keyword-based approach to enhance metadata elements and a syntax-based approach to rectify structural metadata issues, and (3) comparing the outcomes to ensure improved readability and reusability of prepared metadata files.

Guidelines on assigning the subjects of theses and dissertations in repositories

This study aims to investigate how Brazilian institutional repositories guide subject representation in the self-archiving of their information resources. As a method, an exploratory and qualitative study was carried out with the application of an electronic exploratory analysis of repositories and an analysis of transcripts of thesis and dissertation authors’ Individual Verbal Protocols during self-archiving. The analysis of the results in the Brazilian university repository sample shows that 10 have a self-archiving policy and five include guidance to the author on subject representation and metadata quality. The transcription analysis of the authors’ Individual Verbal Protocols reveals that all authors consider keywords important for content representation. The investigation concludes that the self-archiving process is fundamental for the dissemination of scientific information and wide access to the community.

Unsupervised Machine Learning Clustering of Seismic and Infrasound Data Quality Metrics

Abstract Developing techniques for improving quality control (QC) schemes to catch seismic and infrasound data defects continues to be an area of active research. Selecting universal thresholds for the automation of data quality (DQ) checks is an efficient way to find QC issues, but these thresholds may not apply well to multiple stations with varying DQ characteristics. In addition, these thresholds may not catch subtle changes in DQ parameters that still indicate problems. Machine learning can be an alternative way of diagnosing QC issues. K-means clustering, an unsupervised machine learning clustering algorithm, has been effectively used in the past for geophysical pattern exploration. This study furthers k-means applications to DQ analysis through clustering on DQ metrics derived from day-long segments of nuclear explosion monitoring data. Our k-means implementation on broadband seismometer DQ metrics separately clustered mass recenters, calibrations lasting at least one hour, and days without either. Applying this technique to infrasound DQ metrics revealed clusters related to physical issues at the stations, such as missing back volume screws and the flooding of ported pipe inlets. These are both examples of QC issues that are difficult to diagnose or detect through the thresholding of metrics or by inspecting waveforms and spectra. Our results show that k-means clustering can be a useful QC tool in exploring DQ patterns to assist analyst review of station operation and maintenance. The learned knowledge from this exploration can then inform a thresholding workflow on how to tailor to individual stations, or the k-means model could classify data directly.

GEOSCOPE Network: 40 Yr of Global Broadband Seismic Data

Abstract The GEOSCOPE observatory (Institut de physique du globe de Paris [IPGP] and École et Observatoire des Sciences de la Terre de Strasbourg, 1982) provides more than four decades of high-quality continuous broadband data to the scientific community. Started in 1982 with only two stations, the network has grown over the years thanks to numerous international partnerships. At present, 34 stations operate in 18 countries across all continents and on islands throughout the oceans, filling important gaps in global Earth coverage. Most of the first installed stations are still running today, allowing for long-term observations, and new sites are being prospected to further improve global coverage. Over the years, GEOSCOPE has contributed to defining today’s global seismology standards (data format, data quality level, instrumentation requirements), being the French contribution to the international effort for global seismic observations. The stations are instrumented with the best quality seismometers (from the very first STS-1 in the early 80s to the last STS-6A and Trillium T360 today) and digitizers (Q330HR and Centaur) to record with high fidelity the ground motions generated by all types of seismic sources. Real-time data are sent to the tsunami warning centers and both validated and real-time data are available at the IPGP, Epos-France and Earthscope data centers. The quality of GEOSCOPE data and metadata is ensured by daily and yearly validation that enables issue detection and mitigation. GEOSCOPE, in collaboration with the other global networks, has played and continues to play a crucial role in the study of Earth’s structure and global dynamics and the characterization of all types of seismic sources.

Notes onthe data quality ofbibliographic records from the MEDLINE database.

The US National Library of Medicine has created and maintained the PubMed® database, a collection of over 33.8 million records that contain citations and abstracts from the biomedical and life sciences literature. This database is an important resource for researchers and information service providers alike. As part of our work related to the creation of an author graph for coronaviruses, we encountered several data quality issues with records from a curated subset of the PubMed database called MEDLINE. We provide a data quality assessment for records selected from the MEDLINE database and report on several issues ranging from parsing issues (e.g. character encodings and schema definition weaknesses) to low scores for identifiers against several data quality metrics (e.g. completeness, validity and uniqueness). Database URL https://pubmed.ncbi.nlm.nih.gov.

Improving Testing of Deep-learning Systems

We used differential testing to generate test data to improve diversity of data points in the test dataset and then used mutation testing to check the quality of the test data in terms of diversity. Combining differential and mutation testing in this fashion improves mutation score, a test data quality metric, indicating overall improvement in testing effectiveness and quality of the test data when testing deep learning systems.

The impact of data quality monitoring of a multicenter prospective registry of cardiac implantable electronic devices

Data quality monitoring plays a crucial role in multicenter prospective registries. By maintaining high data accuracy, completeness, and consistency, researchers can improve the overall quality and reliability of the registry data, enabling meaningful conclusions and supporting evidence-based decisions. The purpose of the present study was to evaluate data quality metrics (completeness, accuracy, and temporal plausibility) of a Multicenter Registry of Cardiac Implantable Electronic Devices (CIEDs) and to perform a direct data audit of a random sample of records to assess the agreement levels with the source documents. The CIED Registry was a prospective, multicenter, real-world observational study carried out from January 2020 to December 2022 in five designated centers across Sao Paulo, Brazil. We assessed the data quality of the CIED Registry by using two distinct approaches:•Dynamic data monitoring using features of the REDCap (Research Electronic Data Capture) software, including data reports and data quality rules•Direct data audit in which information from a random sample of 10 % of cases from the coordinating center was compared with original source documentsOur findings suggest that the methodological approach applied to the CIED Registry resulted in high data completeness, accuracy, temporal plausibility, and excellent agreement levels with the source documents.

A Crowdsourcing Recommendation Model for Image Annotations in Cultural Heritage Platforms

Cultural heritage is one of many fields that has seen a significant digital transformation in the form of digitization and asset annotations for heritage preservation, inheritance, and dissemination. However, a lack of accurate and descriptive metadata in this field has an impact on the usability and discoverability of digital content, affecting cultural heritage platform visitors and resulting in an unsatisfactory user experience as well as limiting processing capabilities to add new functionalities. Over time, cultural heritage institutions were responsible for providing metadata for their collection items with the help of professionals, which is expensive and requires significant effort and time. In this sense, crowdsourcing can play a significant role in digital transformation or massive data processing, which can be useful for leveraging the crowd and enriching the metadata quality of digital cultural content. This paper focuses on a very important challenge faced by cultural heritage crowdsourcing platforms, which is how to attract users and make such activities enjoyable for them in order to achieve higher-quality annotations. One way to address this is to offer personalized interesting items based on each user preference, rather than making the user experience random and demanding. Thus, we present an image annotation recommendation system for users of cultural heritage platforms. The recommendation system design incorporates various technologies intending to help users in selecting the best matching images for annotations based on their interests and characteristics. Different classification methods were implemented to validate the accuracy of our work on Egyptian heritage.

Fast and efficient identification of anomalous galaxy spectra with neural density estimation

ABSTRACT Current large-scale astrophysical experiments produce unprecedented amounts of rich and diverse data. This creates a growing need for fast and flexible automated data inspection methods. Deep learning algorithms can capture and pick up subtle variations in rich data sets and are fast to apply once trained. Here, we study the applicability of an unsupervised and probabilistic deep learning framework, the probabilistic auto-encoder, to the detection of peculiar objects in galaxy spectra from the SDSS survey. Different to supervised algorithms, this algorithm is not trained to detect a specific feature or type of anomaly, instead it learns the complex and diverse distribution of galaxy spectra from training data and identifies outliers with respect to the learned distribution. We find that the algorithm assigns consistently lower probabilities (higher anomaly score) to spectra that exhibit unusual features. For example, the majority of outliers among quiescent galaxies are E+A galaxies, whose spectra combine features from old and young stellar population. Other identified outliers include LINERs, supernovae, and overlapping objects. Conditional modelling further allows us to incorporate additional information. Namely, we evaluate the probability of an object being anomalous given a certain spectral class, but other information such as metrics of data quality or estimated redshift could be incorporated as well. We make our code publicly available.

On Image Quality Metadata, FAIR in ML, AI-Readiness and Reproducibility: Fish-AIR example

A new science discipline has emerged within the last decade at the intersection of informatics, computer science and biology: Imageomics. Like most other -omics fields, Imageomics also uses emerging technologies to analyze biological data but from the images. One of the most applied data analysis methods for image datasets is Machine Learning (ML). In 2019, we started working on a United States National Science Foundation (NSF) funded project, known as Biology Guided Neural Networks (BGNN) with the purpose of extracting information about biology by using neural networks and biological guidance such as species descriptions, identifications, phylogenetic trees and morphological annotations (Bart et al. 2021). Even though the variety and abundance of biological data is satisfactory for some ML analysis and the data are openly accessible, researchers still spend up to 80% of their time preparing data into a usable, AI-ready format, leaving only 20% for exploration and modeling (Long and Romanoff 2023). For this reason, we have built a dataset composed of digitized fish specimens, taken either directly from collections or from specialized repositories. The range of digital representations we cover is broad and growing, from photographs and radiographs, to CT scans, and even illustrations. We have added new groups of vocabularies to the dataset management system including image quality metadata, extended image metadata and batch metadata. With the image quality metadata and extended image metadata, we aimed to extract information from the digital objects that can possibly help ML scientists in their research with filtering, image processing and object recognition routines. Image quality metadata provides information about objects contained in the image, features and condition of the specimen, and some basic visual properties of the image, while extended image metadata provides information about technical properties of the digital file and the digital multimedia object (Bakış et al. 2021, Karnani et al. 2022, Leipzig et al. 2021, Pepper et al. 2021, Wang et al. 2021) (see details on Fish-AIR vocabulary web page). Batch metadata is used for separating different datasets and facilitates downloading and uploading data in batches with additional batch information and supplementary files. Additional flexibility, built into the database infrastructure using an RDF framework, will enable the system to host different taxonomic groups, which might require new metadata features (Jebbia et al. 2023). By the combination of these features, along with FAIR (Findable, Accessable, Interoperable, Reusable) principles, and reproducibility, we provide Artificial Intelligence Readiness (AIR; Long and Romanoff 2023) to the dataset. Fish-AIR provides an easy-to-access, filtered, annotated and cleaned biological dataset for researchers from different backgrounds and facilitates the integration of biological knowledge based on digitized preserved specimens into ML pipelines. Because of the flexible database infrastructure and addition of new datasets, researchers will also be able to access additional types of data—such as landmarks, specimen outlines, annotated parts, and quality scores—in the near future. Already, the dataset is the largest and most detailed AI-ready fish image dataset with integrated Image Quality Management System (Jebbia et al. 2023, Wang et al. 2021).

Comparing the efficacy of data-driven denoising methods for a multi-echo fMRI acquisition at 7T

In functional magnetic resonance imaging (fMRI) of the brain the measured signal is corrupted by several (e.g. physiological, motion, and thermal) noise sources and depends on the image acquisition. Imaging at ultrahigh field strength is becoming increasingly popular as it offers increased spatial accuracy. The latter is of particular benefit in brainstem neuroimaging given the small cross-sectional area of most nuclei. However, physiological noise scales with field strength in fMRI acquisitions. Although this problem is in part solved by decreasing voxel size, it is clear that adequate physiological denoising is of utmost importance in brainstem-focused fMRI experiments. Multi-echo sequences have been reported to facilitate highly effective denoising through TE-dependence of Blood Oxygen Level Dependent (BOLD) signals, in a denoising method referred to as multi-echo independent component analysis (ME-ICA). It has not been explored previously how ME-ICA compares to other data-driven denoising approaches at ultrahigh field strength. In the current study, we compared the efficacy of several denoising methods, including anatomical component based correction (aCompCor), Automatic Removal of Motion Artifacts (ICA-AROMA) aggressive and non-aggressive options, ME-ICA, and a combination of ME-ICA and aCompCor. We assessed several data quality metrics, including temporal signal-to-noise ratio (tSNR), delta variation signal (DVARS), spectral density of the global signal, functional connectivity and Shannon spectral entropy. Moreover, we looked at the ability of each method to uncouple the global signal and respiration. In line with previous reports at lower field strengths, we demonstrate that after applying ME-ICA, the data is best post-processed in order to remove spatially diffuse noise with a method such as aCompCor. Our findings indicate that ME-ICA combined with aCompCor and the aggressive option of ICA-AROMA are highly effective denoising approaches for multi-echo data acquired at 7T. ME-ICA combined with aCompCor potentially preserves more signal-of-interest as compared to the aggressive option of ICA-AROMA.

ENRICHING LARGE DOCUMENT STORES WITH INTELLIGENT METADATA: A FRAMEWORK FOR EFFECTIVE KNOWLEDGE MANAGEMENT AND APPLIED ANALYTICS

The current paper focuses on a framework for structuring large document stores with the help of intelligent metadata. The described landscape includes a proprietary knowledge graph which ingests millions of concepts from external, third-party data providers and accommodates internal class taxonomies; an NLP service for automated annotation of textual data; an annotations quality control mechanism; tools for knowledge graph ontology and concept management; and an extensive API layer. The authors present an approach they have tested and proved successful in one of the leading media companies in the world, whose media content is a core data asset. The proposed solutions enable content analytics in their proper context and allow explicit and implicit connections between the content and other company data – i.e., user (media content consumer) data. The latter empowers the efficient application of advanced analytical models for searches and recommendations and the implementation of accurate data-driven virtual assistants. The paper advises addressing the metadata quality concerns, which the authors’ extensive practice identifies as an essential prerequisite for applied analytics delivering significant business value.

CM-Explorer: Dissecting Data Ingestion Problems

Data ingestion validation, the task of certifying the quality of continuously collected data, is crucial to ensure trustworthiness of analytics insights. A widely used approach for validating data quality is to specify, either manually or automatically, so-called data unit tests that check whether data quality metrics lie within expected bounds. We employ conditional unit tests based on conditional metrics (CMs) that compute data quality signals over specific parts of the ingestion data and therefore allow for a fine-grained detection of errors. A violated conditional unit test specifies a set of erroneous tuples in a natural way: the subrelation that its CM refers to. Unfortunately, the downside of their fine-grained nature is that violating unit tests are often correlated: a single error in an ingestion batch may cause multiple tests (each referring to different parts of the batch) to fail. The key challenge is therefore to untangle this correlation and filter out the most relevant violated conditional unit tests, i.e., tests that identify a core set of erroneous tuples and act as an explanation for the errors. We present CM-Explorer, a system that supports data stewards in quickly finding the most relevant violated conditional unit tests. The system consists of three components: (1) a graph explorer for visualizing the correlation structure of the violated unit tests; (2) a relation explorer for browsing the tuples selected by conditional unit tests; and, (3) a history explorer to get insight why conditional unit tests are violated. In this paper, we discuss these components and present the different scenarios that we make available for the demonstration.