Metadata Quality Research Articles

Data quality evaluation criteria for systematic review of epidemiology studies under the Toxic Substances Control Act (TSCA): refinement through practical application in risk evaluation

BACKGROUND AND AIM: The 2016 Frank R. Lautenberg Chemical Safety for the 21st Century Act and the 2017 Procedures for Chemical Risk Evaluation under the Amended Toxic Substances Control Act (TSCA) require the U.S. Environmental Protection Agency (EPA) to conduct risk evaluations using the best available science and weight of the scientific evidence. Therefore, EPA has developed systematic review methods including data quality evaluation criteria to evaluate scientific studies in specific disciplines including epidemiology. EPA has aimed to refine these criteria for practical application in risk evaluation of high-priority chemicals under TSCA. METHODS: EPA considered the pros and cons of multiple existing systematic review frameworks and guidance documents and drew upon these methodologies to develop data quality evaluation domains, metrics, and criteria for the systematic review of epidemiology studies under TSCA. EPA refined these domains, metrics, and criteria through practical application to add specificity to increase consistency among evaluators and to harmonize with the evaluation of animal toxicology studies under TSCA. RESULTS: EPA adopted aspects of methods including the National Toxicology Program (NTP) Office of Health Assessment and Translation (OHAT) and EPA’s Integrated Risk Information System (IRIS) methods, but also aimed to be consistent with other disciplines and to address TSCA scientific standards. CONCLUSIONS: EPA has further refined data quality evaluation criteria for systematic review of epidemiology studies under TSCA in response to comments from peer reviewers, the public, the National Academies, and the TSCA Science Advisory Committee on Chemicals (SACC). Systematic review of epidemiology studies is a critical component of protecting human health under TSCA. DISCLAIMER: The views expressed in this presentation are solely those of the authors and do not represent the policies of EPA. Mention of trade names or commercial products should not be interpreted as an endorsement by EPA. KEY WORDS: Systematic Review, Data Quality Evaluation

Learning and Reasoning for Cultural Metadata Quality: Coupling Symbolic AI and Machine Learning over a Semantic Web Knowledge Graph to Support Museum Curators in Improving the Quality of Cultural Metadata and Information Retrieval

This work combines semantic reasoning and machine learning to create tools that allow curators of the visual art collections to identify and correct the annotations of the artwork as well as to improve the relevance of the content-based search results in these collections. The research is based on the Joconde database maintained by the French Ministry of Culture that contains illustrated artwork records from main French public and private museums representing archeological objects, decorative arts, fine arts, historical and scientific documents, etc. The Joconde database includes semantic metadata that describes properties of the artworks and their content. The developed methods create a data pipeline that processes metadata, trains a Convolutional Neural Network image classification model, makes prediction for the entire collection and expands the metadata to be the base for the SPARQL search queries. We developed a set of such queries to identify noise and silence in the human annotations and to search image content with results ranked according to the relevance of the objects quantified by the prediction score provided by the deep learning model. We also developed methods to discover new contextual relationships between the concepts in the metadata by analyzing the contrast between the concepts’ similarities in the Joconde's semantic model and other vocabularies and we tried to improve the model prediction scores based on the semantic relations. Our results show that cross-fertilization between symbolic AI and machine learning can indeed provide the tools to address the challenges of the museum curators work describing the artwork pieces and searching for the relevant images.

Determining pseudo-invariant calibration sites for comparing inter-mission ocean color data

Comparing inter-mission space instrument performance is crucial to accurate satellite measurements which guarantees the quality of ocean color products. However, comparing inter-mission instrument performance is limited by strong dispersion, which clearly originates from the spatial and temporal variability inside the oceanic sampling sites. We designed a novel comprehensive score metric (CSM) to quantitatively identify the candidate pseudo-invariant calibration sites (PICS) for inter-mission comparisons. We calculated the CSM from a year of ocean color and meteorological products from 2018 using a pixel-by-pixel method with a simple average of a temporal meteorological metric, a spatial aerosol metric, a temporal optical metric, a spatial optical metric, a data quality metric, a spectral shape metric, and a directional homogeneity metric. When we filtered the data with the threshold CSM > 0.6, we found that atmospheric and oceanic conditions from two smooth belts in the low latitude open ocean were more clear, stable, and homogeneous than other regions, and we suggested these smooth belts as candidate PICS. With image data from the Visible Infrared Imaging Radiometer (VIIRS) and Medium Resolution Spectral Imager II (MERSI II), we found that our candidate PICS were more effective than regions with low CSM in providing stable synchronous data for inter-mission comparisons. The MERSI II instrument, however, experienced significant degradation in radiance measurements from January to April in 2021, while the VIIRS instrument performed well. These results suggested that the CSM values were “experimental” but, under restrictive conditions, were sufficient for an inter-mission comparison and calibration application.

Evaluating Metadata Quality in LGBTQ+ Digital Community Archives

This project evaluated metadata in digital LGBTQ+ community archives in order to determine its quality and how metadata quality effects the sustainability of digital community archives. This project uses a case study approach, using content analysis to evaluate metadata quality of three LGBTQ+ digital archives: Transas City, The History Project, and ONE Archives. Analysis found that the metadata in LGBTQ+ digital community archives is inconsistent and often only meets the minimum requirements for quality metadata. Further, this study concluded that professional guidelines and practices for metadata strip away the personality and uniqueness that is key to community archives success and purpose.

Law Reviews, Open Metadata and RSS Feeds

AbstractMost scientific publications have their metadata available as freely accessible and machine-readable information at CrossRef. However, student-edited law reviews have not followed suit with this practice. Consequently, a large part of legal research remains in a blind spot for scientometric analyses and tools. The present paper, by Andreas Nishikawa-Pacher, investigates whether the law reviews’ RSS feeds could serve as equivalent sources for open scholarly metadata. The suitability of RSS feeds from 51 student-edited law reviews (as indexed in Web of Science's Social Science Citation Index) was assessed with regards to three fictitious meta-scientific applications – namely (1) a ‘latest paper’-tool that lists the law reviews’ newest publications with links, abstracts and dates, (2) an author database and (3) a calculation of the mutual citation counts among different law reviews. This paper finds that only 21 law reviews offer functional RSS feeds, and while they were suitable for a basic ‘latest papers’-tool, due to their low metadata quality they cannot aid in generating an author database or in counting the mutual citations among law reviews. The result suggests that law reviews would be advised to adopt digital object identifiers (DOIs) and start depositing openly accessible metadata, for otherwise their scholarly impact and visibility will further decline.

Describing Data in Image Format: Proposal of a Metadata Model and Controlled Vocabularies

Research data management (RDM) includes people with different needs, specific scientific contexts, and diverse requirements. The description is a big challenge in the domain of RDM. Metadata plays an essential role, allowing the inclusion of essential information for the interpretation of data, enhances the reuse of data and its preservation. The establishment of metadata models can facilitate the process of description and contribute to an improvement in the quality of metadata. When we talk about image data, the task is even more difficult, as there are no explicit recommendations to guide image management. In this work, we present a proposal for a metadata model for image description. To validate the model, we followed an experiment of data description, where eleven participants described images from their research projects, using a metadata model proposed. The experiment shows that participants do not have formal practices for describing their imagery data. Yet, they provided valuable contributions and recommendations to the final definition of a metadata model for image description, to date nonexistent. We also developed controlled vocabularies for some descriptors. These vocabularies aim to improve the image description process, facilitate metadata model interpretation, and reduce the time and effort devoted to data description.

Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research

BackgroundIdentifying and removing reference duplicates when conducting systematic reviews (SRs) remain a major, time-consuming issue for authors who manually check for duplicates using built-in features in citation managers. To address issues related to manual deduplication, we developed an automated, efficient, and rapid artificial intelligence-based algorithm named Deduklick. Deduklick combines natural language processing algorithms with a set of rules created by expert information specialists. MethodsDeduklick’s deduplication uses a multistep algorithm of data normalization, calculates a similarity score, and identifies unique and duplicate references based on metadata fields, such as title, authors, journal, DOI, year, issue, volume, and page number range. We measured and compared Deduklick’s capacity to accurately detect duplicates with the information specialists’ standard, manual duplicate removal process using EndNote on eight existing heterogeneous datasets. Using a sensitivity analysis, we manually cross-compared the efficiency and noise of both methods.DiscussionDeduklick achieved average recall of 99.51%, average precision of 100.00%, and average F1 score of 99.75%. In contrast, the manual deduplication process achieved average recall of 88.65%, average precision of 99.95%, and average F1 score of 91.98%. Deduklick achieved equal to higher expert-level performance on duplicate removal. It also preserved high metadata quality and drastically reduced time spent on analysis. Deduklick represents an efficient, transparent, ergonomic, and time-saving solution for identifying and removing duplicates in SRs searches. Deduklick could therefore simplify SRs production and represent important advantages for scientists, including saving time, increasing accuracy, reducing costs, and contributing to quality SRs.

Analysis of SLAM-Based Lidar Data Quality Metrics for Geotechnical Underground Monitoring

Adverse ground behavior events, such as convergence and ground falls, pose critical risks to underground mine safety and productivity. Today, monitoring of such failures is primarily conducted using legacy techniques with low spatial and temporal resolution while exposing workers to hazardous environments. This study assesses the potential of novel simultaneous localization and mapping (SLAM)-based light detection and ranging (Lidar) data quality for rapid, digital, and eventually autonomous mine-wide underground geotechnical monitoring. We derive a comprehensive suite of quality metrics based on tests in two underground mines for two state-of-the-art mobile laser scanning (MLS) systems. Our results provide evidence that SLAM-based MLS provides data of the quality required to detect geotechnically relevant changes while being significantly more efficient for large mine layouts when compared to traditional static systems. Additionally, we show that SLAM-specific processing can achieve an order of magnitude better relative accuracy relevant for change detection than quality metrics derived from traditionally deployed tests would suggest while reducing SLAM drift error by up to 90%. In collaboration with an operating block cave mine, we confirm these capabilities in field tests on a mine-wide scale and, for the first time, demonstrate methods of rockfall detection using MLS data. While more work is required to investigate optimal collection, processing, and utilization of MLS data, we demonstrate its potential to become an effective and widely applicable data source for rapid, accurate, and comprehensive geotechnical inspections.

OGD metadata country portal publishing guidelines compliance: A multi-case study search for completeness and consistency

To guarantee a certain level of data usability, metadata which is information about the OGD, is needed. For this reason, OGD administrators provide metadata publishing guidelines to request, among other things, that certain metadata field always be present (completeness) and that field validation rules, when provided, be met (consistency). Lack of compliance of OGD metadata with publishing guidelines in country portals constitutes poor metadata quality control and will drastically decrease the effective use/reuse of the data and keep the expected benefits from being realized. This study explores the extent of metadata compliance, in terms of completeness and consistency, of three OGD leading country portals (Canada, USA, New Zealand) with respect to their own metadata publishing guidelines. A statistical comparison of population proportions among the countries was performed. The results indicate that the compliance of the USA is far less than Canada and New Zealand in terms of datasets with required fields set (completeness). On the other hand, New Zealand trails Canada and the USA in terms of OGD datasets with valid (consistent) fields. These results indicate that OGD metadata compliance is rather poor, suggesting insufficient quality control. Finally, this study concludes with suggestions to improve the OGD platforms so that metadata compliance can be automatically enforced.

The HAPPE plus Event-Related (HAPPE+ER) software: A standardized preprocessing pipeline for event-related potential analyses

Event-Related Potential (ERP) designs are a common method for interrogating neurocognitive function with electroencephalography (EEG). However, the traditional method of preprocessing ERP data is manual-editing – a subjective, time-consuming processes. A number of automated pipelines have recently been created to address the need for standardization, automation, and quantification of EEG data pre-processing; however, few are optimized for ERP analyses (especially in developmental or clinical populations). We propose and validate the HAPPE plus Event-Related (HAPPE+ER) software, a standardized and automated pre-processing pipeline optimized for ERP analyses across the lifespan. HAPPE+ER processes event-related potential data from raw files through preprocessing and generation of event-related potentials for statistical analyses. HAPPE+ER also includes post-processing reports of both data quality and pipeline quality metrics to facilitate the evaluation and reporting of data processing in a standardized manner. Finally, HAPPE+ER includes post-processing scripts to facilitate validating HAPPE+ER performance and/or comparing to performance of other preprocessing pipelines in users’ own data via simulated ERPs. We describe multiple approaches with simulated and real ERP data to optimize pipeline performance and compare to other methods and pipelines. HAPPE+ER software is freely available under the terms of GNU General Public License at https://www.gnu.org/licenses/#GPL

One Concept, One Term, Good Practice but How to Achieve? —Improving Facet Values Quality for Samuel Proctor Oral History Collection, Hosted by the University of Florida Digital Collections

Faceted search, also known as dynamic taxonomies, is a popular feature applied to digital collection sites. Appearing as clickable labels, facets facilitate search result refinement and content browsing. To achieve the utmost efficiency, faceted search requires each facet value to represent a single concept–that is, one controlled vocabulary term represents one concept. However, in reality, this status is hard to achieve. An example of this can be seen in the digital collection of Samuel Proctor Oral History Program, a high-profile international resource hosted by the University of Florida Digital Collections. The Topical Subject and Genre terms appear with many vocabulary control issues: the same concept is often expressed with different terms; the same term appears with different spelling variations; and/or outdated terms mingle with more up-to-date ones. Additionally, compound terms that represent multiple concepts prohibit the grouping of content that share individual concepts. In short, a great deal of improvement will be needed to optimize the faceted search. To address these issues, the Digital Support Services department of the George A. Smathers Libraries at the University of Florida launched a pilot metadata remediation project using an out-of-box product – Oxygen XML Editor. This article, in addition to providing one more metadata remediation case study, traces the discussions around metadata quality and analyses the general metadata remediation process. Moreover, this article enriches the discussion of vocabulary control in relation to a core function of digital collection sites–faceted search.

Factors affecting data quality of online questionnaires: Issues and metrics for sensory and consumer research

Methodology used in sensory and consumer research has changed rapidly over the past several decades, and consumer research is now frequently executed online. This comes with advantages, but also with the potential for poor quality data due to uncontrolled elements of survey administration and execution. Published papers that utilize online data rarely report metrics of data quality or note precautions that were taken to ensure valid data. The aim of this paper is to raise awareness of the factors that influence online data quality. This is achieved by 1) identifying factors that can impact the reliability and validity of consumer data obtained with online questionnaires, 2) highlighting indices of online questionnaire data quality that can be used by researchers to assess the likelihood that their data are of poor quality, and 3) recommending a number of indices, counter-measures and best practices that can be used by sensory and consumer researchers to assess online questionnaire data quality and to ensure the highest degree of reliability and validity of their data. By making researchers aware of these sources of invalidity and by presenting available remedies and good practices, it is our intention that sensory and consumer scientists will adopt these measures and report them as a routine part of publishing online questionnaire data. Specific suggestions are offered regarding the important elements of data quality that should be included in scientific manuscripts.

A focus groups study on data sharing and research data management

Data sharing can accelerate scientific discovery while increasing return on investment beyond the researcher or group that produced them. Data repositories enable data sharing and preservation over the long term, but little is known about scientists’ perceptions of them and their perspectives on data management and sharing practices. Using focus groups with scientists from five disciplines (atmospheric and earth science, computer science, chemistry, ecology, and neuroscience), we asked questions about data management to lead into a discussion of what features they think are necessary to include in data repository systems and services to help them implement the data sharing and preservation parts of their data management plans. Participants identified metadata quality control and training as problem areas in data management. Additionally, participants discussed several desired repository features, including: metadata control, data traceability, security, stable infrastructure, and data use restrictions. We present their desired repository features as a rubric for the research community to encourage repository utilization. Future directions for research are discussed.

A method to produce metadata describing and assessing the quality of spatial landmark datasets in mountain area

Abstract. The increase of recreational activities in the mountains and a growing amount of websites proposing geographic data, offer new opportunities for societal needs such as mountain rescue, biodiversity monitoring, outdoor activities. However, the main issue with the websites data is the lack of metadata that minimizes its reuse outside the community that produced the data. The goal of this paper is to study and generate quality and descriptive metadata using ISO standards. To this end, we propose a method based on a common vocabulary such as an ontology and a data matching process. The first one allows to associate to each type of feature from an available geographic dataset an ontology class that will facilitate data matching, reproducibility of results and minimize semantic heterogeneity. The second one allows to define matching links between features representing the same entity in the real world and compute quality indicators based on the validated links. Finally, at the end of this process, we are able to generate descriptive and quality metadata. By following ISO standards and using the QualityML dictionary for measures, the metadata is serialized to XML and can finally be published as open source. Our approach was applied to five different landmark datasets in the French Alps region. New insights were acquired regarding positional accuracy and semantic granularity.

8. Optical genome mapping analysis of FMR1 expansions in fragile X syndrome

Fragile X syndrome (FXS) is associated with cognitive impairment and intellectual disability. Nearly all cases of FXS are associated with an expansion of a CGG triplet repeat in the FMR1 gene. Phenotype severity is often correlated with the expansion size; thus, accurate sizing of the CCG repeat array is crucial. The repetitive nature of these regions presents difficulties: 1. PCR is unable to traverse through long repeats; 2. sequencing has limited read lengths; 3. Southern blot may be inaccurate, time consuming, and expensive. Optical genome mapping (OGM) has the potential to address some of these shortcomings. OGM utilizes images of ultra-long DNA molecules, labeled at specific sequence motifs linearized in nanochannel arrays and can be used for structural and copy number variation calling and absence of heterozygosity analysis. Bionano Genomics has developed a targeted analysis workflow for analysis of FMR1. To evaluate the capability of measuring repeat arrays consistent with normal (<45 repeats), premutation (55-200 repeats), and full mutation (>200 repeats), we analyzed the FMR1 repeats in 75 samples. We also analyzed the repeat region on 20 phenotypically normal control subjects. We observed alleles consistent with annotation across the entire range of repeat counts. Analytical sensitivity was measured at 97% with 100% PPV for expansions >200 repeats. The largest expansion we detected was ∼1000 repeats. In the control samples, we measured repeat counts below the full mutation cutoff in all cases. Repeatability studies were carried out to show analytical consistency and to establish minimum molecule quality requirements. EnFocus™ analysis report provides pass/fail for QC metrics for input data quality as well as analytic measurement quality using internal control regions of the genome on each autosome. OGM performance for FMR1 repeat lengths analyzed here show a much higher dynamic range compared to PCR and higher precision compared to Southern blot.

Globally Accessible Distributed Data Sharing (GADDS): a decentralized FAIR platform to facilitate data sharing in the life sciences.

MotivationTechnical advances have revolutionized the life sciences and researchers commonly face challenges associated with handling large amounts of heterogeneous digital data. The Findable, Accessible, Interoperable and Reusable (FAIR) principles provide a framework to support effective data management. However, implementing this framework is beyond the means of most researchers in terms of resources and expertise, requiring awareness of metadata, policies, community agreements and other factors such as vocabularies and ontologies.ResultsWe have developed the Globally Accessible Distributed Data Sharing (GADDS) platform to facilitate FAIR-like data-sharing in cross-disciplinary research collaborations. The platform consists of (i) a blockchain-based metadata quality control system, (ii) a private cloud-like storage system and (iii) a version control system. GADDS is built with containerized technologies, providing minimal hardware standards and easing scalability, and offers decentralized trust via transparency of metadata, facilitating data exchange and collaboration. As a use case, we provide an example implementation in engineered living material technology within the Hybrid Technology Hub at the University of Oslo.Availability and implementationDemo version available at https://github.com/pavelvazquez/GADDS.Supplementary information Supplementary data are available at Bioinformatics online.

Greenland Geothermal Heat Flow Database and Map (Version 1)

Abstract. We compile and analyze all available geothermal heat flow measurements collected in and around Greenland into a new database of 419 sites and generate an accompanying spatial map. This database includes 290 sites previously reported by the International Heat Flow Commission (IHFC), for which we now standardize measurement and metadata quality. This database also includes 129 new sites, which have not been previously reported by the IHFC. These new sites consist of 88 offshore measurements and 41 onshore measurements, of which 24 are subglacial. We employ machine learning to synthesize these in situ measurements into a gridded geothermal heat flow model that is consistent across both continental and marine areas in and around Greenland. This model has a native horizontal resolution of 55 km. In comparison to five existing Greenland geothermal heat flow models, our model has the lowest mean geothermal heat flow for Greenland onshore areas. Our modeled heat flow in central North Greenland is highly sensitive to whether the NGRIP (North GReenland Ice core Project) elevated heat flow anomaly is included in the training dataset. Our model's most distinctive spatial feature is pronounced low geothermal heat flow (&lt; 40 mW m−2) across the North Atlantic Craton of southern Greenland. Crucially, our model does not show an area of elevated heat flow that might be interpreted as remnant from the Icelandic plume track. Finally, we discuss the substantial influence of paleoclimatic and other corrections on geothermal heat flow measurements in Greenland. The in situ measurement database and gridded heat flow model, as well as other supporting materials, are freely available from the GEUS Dataverse (https://doi.org/10.22008/FK2/F9P03L; Colgan and Wansing, 2021).

A Survey of Data Quality Measurement and Monitoring Tools

High-quality data is key to interpretable and trustworthy data analytics and the basis for meaningful data-driven decisions. In practical scenarios, data quality is typically associated with data preprocessing, profiling, and cleansing for subsequent tasks like data integration or data analytics. However, from a scientific perspective, a lot of research has been published about the measurement (i.e., the detection) of data quality issues and different generally applicable data quality dimensions and metrics have been discussed. In this work, we close the gap between data quality research and practical implementations with a detailed investigation on how data quality measurement and monitoring concepts are implemented in state-of-the-art tools. For the first time and in contrast to all existing data quality tool surveys, we conducted a systematic search, in which we identified 667 software tools dedicated to “data quality.” To evaluate the tools, we compiled a requirements catalog with three functionality areas: (1) data profiling, (2) data quality measurement in terms of metrics, and (3) automated data quality monitoring. Using a set of predefined exclusion criteria, we selected 13 tools (8 commercial and 5 open-source tools) that provide the investigated features and are not limited to a specific domain for detailed investigation. On the one hand, this survey allows a critical discussion of concepts that are widely accepted in research, but hardly implemented in any tool observed, for example, generally applicable data quality metrics. On the other hand, it reveals potential for functional enhancement of data quality tools and supports practitioners in the selection of appropriate tools for a given use case.

Extensive evaluation of ATAC-seq protocols for native or formaldehyde-fixed nuclei

BackgroundThe “Assay for Transposase Accessible Chromatin sequencing” (ATAC-seq) is an efficient and easy to implement protocol to measure chromatin accessibility that has been widely used in multiple applications studying gene regulation. While several modifications or variants of the protocol have been published since it was first described, there has not yet been an extensive evaluation of the effects of specific protocol choices head-to-head in a consistent experimental setting. In this study, we tested multiple protocol options for major ATAC-seq components (including three reaction buffers, two reaction temperatures, two enzyme sources, and the use of either native or fixed nuclei) in a well-characterized cell line. With all possible combinations of components, we created 24 experimental conditions with four replicates for each (a total of 96 samples). In addition, we tested the 12 native conditions in a primary sample type (mouse lung tissue) with two different input amounts. Through these extensive comparisons, we were able to observe the effect of different ATAC-seq conditions on data quality and to examine the utility and potential redundancy of various quality metrics.ResultsIn general, native samples yielded more peaks (particularly at loci not overlapping transcription start sites) than fixed samples, and the temperature at which the enzymatic reaction was carried out had a major impact on data quality metrics for both fixed and native nuclei. However, the effect of various conditions tested was not always consistent between the native and fixed samples. For example, the Nextera and Omni buffers were largely interchangeable across all other conditions, while the THS buffer resulted in markedly different profiles in native samples. In-house and commercial enzymes performed similarly.ConclusionsWe found that the relationship between commonly used measures of library quality differed across temperature and fixation, and so evaluating multiple metrics in assessing the quality of a sample is recommended. Notably, we also found that these choices can bias the functional class of elements profiled and so we recommend evaluating several formulations in any new experiments. Finally, we hope the ATAC-seq workflow formulated in this study on crosslinked samples will help to profile archival clinical specimens.

A data-driven workflow to improve energy efficient operation of commercial buildings: A review with real-world examples

Data-driven building operation and maintenance research such as metadata inference, fault detection and diagnosis, occupant-centric controls (OCCs), and non-invasive load monitoring have emerged (NILM) as independent domains of study. However, there are strong dependencies between these domains; for example, quality of metadata affects the usability of fault detection and diagnostics techniques. Further, faults in controls hardware and programs limit the performance of OCCs. To this end, a literature review was conducted to identify the dependencies between these domains of research. Additionally, real-world examples using operational data from three institutional buildings in Ottawa, Canada, were provided and discussed to demonstrate these dependencies. Finally, a holistic tool-agnostic workflow was introduced which suggested the implementation of operational energy efficiency measures in the following order to ensure their full potential: (1) improve metadata, (2) address faults, (3) implement OCCs, and (4) monitor enhanced key performance indicators (KPIs). The proposed workflow is intended to be comprehensive, reproducible, nonintrusive, and inexpensive to implement. Practical applications: Optimization of building operations has been emerging among energy management professionals as a relatively low-cost means to achieve energy efficiency and minimize occupants’ discomfort. To this end, this study introduces a tool-agnostic data-driven workflow to building energy management practitioners that can assist them in achieving increased energy efficiency. The proposed workflow recognizes the interdependency of the various domains of research which have historically been treated independently.