Data Ingestion Research Articles

Implementing an open-source sensor data ingestion, fusion, and analysis capabilities for smart manufacturing

In many modern enterprises, factory managers monitor their machinery and processes to prevent faults and product defects, and maximize the productivity and efficiency. Asset condition, product quality and system productivity monitoring consume some 40-70% of the production costs. Oftentimes, resource constraints have prevented the adoption and implementation of these practices in small businesses. Recent evolution of manufacturing-as-a-service and increased digitalization opens opportunities for small and medium scale companies to adopt smart manufacturing practices, and thereby surmount these constraints. Specifically, sensor wrappers that delineate the specifications of sensor integration into manufacturing machinery, with appropriate edge-cloud computing and communication architecture can provide even small businesses with a real-time data pipeline to monitor their manufacturing machines. However, the data in itself is difficult to interpret locally. Additionally, proprietary standards and products of the various components of a sensor wrapper make it difficult to implement a sensor wrapper schema. In this paper, we report an open-source method to integrate sensors into legacy manufacturing equipment and hardware. We had implemented this pipeline with off-the-shelf sensors to a polisher (from Buehler), a shaft grinding machine (from Micromatic), and a hybrid manufacturing machine (from Optomec), and used hardware and software components such as a National Instruments Data Acquisition (NI-DAQ) module to collect and stream live data. We evaluate the performance of the data pipeline as it connects to the Smart Manufacturing Innovation Platform (SMIP)—web-based data ingestion platform part of the Clean Energy Smart Manufacturing Innovation Institute (CESMII), a U.S. Department of Energy-sponsored initiative—in terms of data volume versus latency tradeoffs. We demonstrate a viable implementation of Smart Manufacturing by creating a vendor-agnostic web dashboard that fuses multiple sensors to perform real-time performance analysis with lossless data integrity.

A machine learning model for predicting, diagnosing, and mitigating health disparities in hospital readmission

The management of hyperglycemia in hospitalized patients has a significant impact on both morbidity and mortality. Therefore, it is important to predict the need for diabetic patients to be hospitalized. However, using standard machine learning approaches to make these predictions may result in health disparities caused by biases in the data related to social determinants (such as race, age, and gender). These biases must be removed early in the data collection process, before they enter the system and are reinforced by model predictions, resulting in biases in the model’s decisions. In this paper, we propose a machine learning pipeline capable of making predictions as well as detecting and mitigating biases in the data and model predictions. This pipeline analyses the clinical data and determines whether biases exist in the data, if so, it removes those biases before making predictions. We evaluate the performance of the proposed method on a clinical dataset using accuracy and fairness measures. The findings of the results show that when we mitigate biases early during the data ingestion, we get fairer predictions.

Growing pains of a data repository: GRIIDC's evolution from environmental disaster rapid response to promoting FAIR data

GRIIDC is a multidisciplinary data repository created in the aftermath of the Deepwater Horizon oil spill. Development of the repository occurred even as researchers collected post-spill data, and as a result, the data management system initially focused on the ingestion of data and metadata. Data sharing was not as prevalent as it is currently, and many researchers were not familiar with data sharing and data organization best practices. Implementation of data management planning, submission, citation, and distribution features required many iterations and occurred while GRIIDC was assisting researchers with managing their rapid response data. From this challenging beginning, over the decade since the Deepwater Horizon oil spill, GRIIDC has improved the data management system and the training of researchers, which has enhanced the ease of submission and quality of data submitted. The GRIIDC system has also evolved to prioritize the implementation of FAIR data principles to ensure the data are findable, accessible, interoperable, and reusable. All data are issued digital object identifiers (DOIs) through DataCite and are findable via GRIIDC's data search page, DataONE, and Google Dataset Search. Each dataset has a landing page where the data and metadata can be accessed. GRIIDC is continuously striving to add FAIR principles to the system. Although there are still many challenges including quality of data and metadata received, funding limitations, and program priorities, GRIIDC must always continue to improve its ability to meet user needs while implementing FAIR data principles.

Accelerate the Creation of the cross agency Human Services Dataset.

ObjectiveDevelop a digital solution for automated data ingestion and rapid update of the large-scale Human Services Dataset (HSDS) which brings together data from across government to take a powerful view of the service usage to improve outcomes of communities. ApproachThe Centre for Health Record Linkage (CHeReL) hosts a secure, high-performing data linkage system, including a Master Linkage Key (MLK) of administrative health datasets, and generates linked data to inform policy decisions. Since 2018, CHeReL has also been annually linking over 70 frontline datasets to create a large-scale longitudinal linked dataset of over 2.5 billion records. Over the course of 2021, the CHeReL led a project to incrementally improve the currency of the HSDS in compressed timeframes. This provided opportunity to assess value and feasibility of more frequent updates to the dataset within the evaluation and investment context. ResultsThe automated data Ingestion and validation led to a significant reduction in the data processing timeframes for the Accelerated linkage. We observed 80% reduction in Data ingestion and 75% reduction in data validation. The digital solution also allows asset owners to register and approve new data providers, monitor their data provision in real-time and report on data sourcing. This provides transparency to the Asset Owner and reduces the need for time-intensive and manual processes to jointly monitor data provision with the Data Linkage Centre. The digital solution also has the capability to support Data Providers automate their data feeds and provide on a regular basis through a secure non- touch process. This reduces on-going workload and ensures on-time provision. ConclusionThe process requires a systematic change in the upstream data source, and we requested participating agencies to send us data in an agreed format. The receipt of files in standard format is pivotal for reducing the overall timeframes of HSDS creation and leverage it for policy and investment purpose.

Deployment of a Free-Text Analytics Platform at a UK National Health Service Research Hospital: CogStack at University College London Hospitals.

BackgroundAs more health care organizations transition to using electronic health record (EHR) systems, it is important for these organizations to maximize the secondary use of their data to support service improvement and clinical research. These organizations will find it challenging to have systems capable of harnessing the unstructured data fields in the record (clinical notes, letters, etc) and more practically have such systems interact with all of the hospital data systems (legacy and current).ObjectiveWe describe the deployment of the EHR interfacing information extraction and retrieval platform CogStack at University College London Hospitals (UCLH).MethodsAt UCLH, we have deployed the CogStack platform, an information retrieval platform with natural language processing capabilities. The platform addresses the problem of data ingestion and harmonization from multiple data sources using the Apache NiFi module for managing complex data flows. The platform also facilitates the extraction of structured data from free-text records through use of the MedCAT natural language processing library. Finally, data science tools are made available to support data scientists and the development of downstream applications dependent upon data ingested and analyzed by CogStack.ResultsThe platform has been deployed at the hospital, and in particular, it has facilitated a number of research and service evaluation projects. To date, we have processed over 30 million records, and the insights produced from CogStack have informed a number of clinical research use cases at the hospital.ConclusionsThe CogStack platform can be configured to handle the data ingestion and harmonization challenges faced by a hospital. More importantly, the platform enables the hospital to unlock important clinical information from the unstructured portion of the record using natural language processing technology.

A Machine-Learning Pipeline for Large-Scale Power-Quality Forecasting in the Mexican Distribution Grid

Electric power distribution networks face increasing factors for power-quality (PQ) deterioration, such as distributed, renewable-energy generation units and countless high-end electronic devices loaded as controllers or in standalone mode. Consequently, government regulations are issued worldwide to set up strict PQ distribution standards; the distribution grids must comply with those regulations. This situation drives research towards PQ forecasting as a crucial part of early-warning systems. However, most of the approaches in the literature disregard the big-data nature of the problem by working on small datasets. These datasets come from short-scale off-grid configurations or selected portions of a larger power grid. This article addresses a study case from a region-sized state-owned Mexican distribution grid, where the company must preserve essential PQ standards in approximately 700 distribution circuits and 150 quality-control nodes. We implemented a machine-learning pipeline with nearly 4000 univariate forecasting models to address this challenge. The system executes a weekly forecasting pipeline and daily data ingestion and preprocessing pipeline, processing massive amounts of data ingested. The implemented system, MIRD (an acronym for Monitoreo Inteligente de Redes de Distribution—Intelligent Monitoring of Distribution Networks), is an unprecedented effort in the production, deployment, and continuous use of forecasting models for PQ indices monitoring. To the extent of the authors’ best knowledge, there is no similar work of this type in any other Latin-American distribution grid.

Growth and Collaboration in the Atlas of Living Australia

Over the past three years, the Atlas of Living Australia (ALA), a data infrastructure in its own right and the Australian node of the Global Biodiversity Information Facility (GBIF), has been both enjoying both rapid growth (now with over 50 employees covering a broad project portfolio), and maturing as an organisation with formal processes in place for strategy and workplan development, project management and planning. Having recently upgraded our core infrastructure to align with GBIF and now sharing a code base for our data ingestion pipelines, we have been working on extending the occurrence data model with a focus on ecological survey sites and events and will continue to participate in the development of GBIF's diversified data model. We have been developing a framework with our jurisdictional agencies for managing restricted data, which will see us develop our Sensitive Data Service to handle a broader range of data sensitivity scenarios than the current system, which obfuscates by species rules only, whereas there are requirements to manage sensitivity at the levels of dataset, provider or location. We have begun a cross-agency project for aggregating genomic data for Australian species from multiple platforms, and a new technical roadmap will guide us as we redevelop and modernise our suite of web services and applications. This presentation will give a very brief overview of our current major project work, relating it to our stated strategies to deliver trusted data and robust services, partner for impact, and support decision making.

Share the tensor tea

We demonstrate Tensor Query Processor (TQP): a query processor that automatically compiles relational operators into tensor programs. By leveraging tensor runtimes such as PyTorch, TQP is able to: (1) integrate with ML tools (e.g., Pandas for data ingestion, Tensorboard for visualization); (2) target different hardware (e.g., CPU, GPU) and software (e.g., browser) backends; and (3) end-to-end accelerate queries containing both relational and ML operators. TQP is generic enough to supports the TPC-H benchmark, and it provides performance that are comparable to, and often better than, that of specialized CPU and GPU query processors.

Virtual Industrial Process Optimization Models with Cloud Management

The increasing development in ICT has driven and accelerated the advent of IoT technologies, creating the opportunity to empower and innovate data management and production management within factories with data collection, integration, storage and processing as a key challenge. By using the technologies offered by the fourth industrial revolution, it has been possible to test models through simulation before implementing them in real physical systems and with greater complexity. Specifically, by using a combination of cloud computing, IoT platforms, programming software and calculation software, three industrial processes have been virtualized with protocols based on the TCP/IP standard. By considering relevant parameters and variables, an optimization model is developed to improve the operation sequences reducing costs within the factory and a second model to minimize the total cost of operation subject to power balance and production limits. An indispensable element to consider a 4.0 architecture is the database. Mongodb Atlas NoSQL has been able to provide the research with both effective data ingestion and retrieval, which are challenging tasks in case of opting for a SQL database. This paper focuses on the concept of optimization models and their data management from the cloud. The results show that Mongodb is a highly reliable database with enormous scalability and reliability to store data coming from mathematical models or sensors of virtualized production processes, with or without a predefined structure. Moreover, with IoT platforms, it is possible to visualize the retrieved data and interpret them in a graphical way to improve a decision making within the processes.

PROMENADE: A big data platform for handling city complex networks with dynamic graphs

Continuous data streams, generated by modern sensed cities, open many opportunities and perspectives in terms of developing new innovative services. To exploit this potential, flexible and scalable platforms are needed to ease the design, development, deployment, and operations of new city services. In recent years, several problem-specific platforms have been proposed in different application domains; however, to boost the evolution of smart cities, we claim the need for city-oriented platforms that can be easily customized to address different day-to-day life challenging problems. In this paper, we present the main architectural challenges and solutions proposed for the design of a novel open-source platform (named PROMENADE) characterized by: i) a data-driven graph-based modeling support to ensure high generality for addressing disparate problems related to the networked nature of many city infrastructures and systems, ii) the dynamic nature of the graph entities updated in real-time from different sources (e.g., IoT/Edge networks, data providers, etc.), and iii) high efficiency, scalability and flexibility to easily support new city services. The platform is designed around a general-purpose core that provides a set of built-in standard features such as data ingestion, storage, processing, and visualization exposed as a collection of containerized microservices. A specialization of the platform has been developed for road networks monitoring. It has been deployed in OpenShift/Kubernetes and tested using realistic datasets collected from the city of Lyon, France. The analysis addresses an important problem of big data processing pipelines: the synchronization between data ingestion and processing in order to produce an accurate result in useful time. To this end, we study different approaches for synchronization and show how the end-to-end latency is kept under control by leveraging the scalability of the platform.

EQIP's First Year: A Step Closer to Higher Quality in Surgical Education

To describe the first year of the Educational Quality Improvement Program (EQIP) DESIGN: The Educational Quality Improvement Program (EQIP) was formed by the Association of Program Directors in Surgery (APDS) in 2018 as a continuous educational quality improvement program. Over 18 months, thirteen discrete goals for the establishment of EQIP were refined and executed through a collaborative effort involving leaders in surgical education. Alpha and beta pilots were conducted to refine the data queries and collection processes. A highly-secure, doubly-deidentified database was created for the ingestion of resident and program data. 36 surgical training programs with 1264 trainees and 1500 faculty members were included in the dataset. 51,516 ERAS applications to programs were also included. Uni- and multi-variable analysis was then conducted. EQIP was successfully deployed within the timeline described in 2020. Data from the ACGME, ABS, and ERAS were merged with manually entered data by programs and successfully ingested into the EQIP database. Interactive dashboards have been constructed for use by programs to compare to the national cohort. Risk-adjusted multivariable analysis suggests that increased time in a technical skills lab was associated with increased success on the ABS's Qualifying Examination, alone. Increased time in a technical skills lab and the presence of a formal teaching curriculum were associated with increased success on both the ABS's Qualifying and Certifying Examination. Program type may be of some consequence in predicting success on the Qualifying Examination. The APDS has proved the concept that a highly secure database for the purpose of continuous risk-adjusted quality improvement in surgical education can be successfully deployed. EQIP will continue to improve and hopes to include an increasing number of programs as the barriers to participation are overcome.

Building a 360-Degree Customer View with Salesforce Data Cloud

Attaining a 360-degree view of a customer is crucial for any contemporary organization which is seeking to improve the customer touchpoints and make better decisions. Salesforce Data Cloud helps organizations by amalgamating and processing data from different sources to form a single, usable customer profile. This paper aims at discussing methodology of creating a 360-degree customer view using the Salesforce Data Cloud platform, where the main focus is paid to the intricate data ingestion processes, transformation, warehousing, and analysis. It also offers a step by step A to Z analysis of the structure of the system with clear flow charts, pseudocodes and diagrams depicting the stages of data flow. Furthermore, the paper provides insights into future directions such as incorporation of AI/ML, enhanced data security and real-time processing of data. Through the use of Salesforce Data Cloud, all the entities involved can benefit from credible information that can enhance not only customer engagement, but also bolster decision making processes

Smart connected worker edge platform for smart manufacturing: Part 1—Architecture and platform design

AbstractThe challenge of sustainably producing goods and services for healthy living on a healthy planet requires simultaneous consideration of economic, societal, and environmental dimensions in manufacturing. Enabling technology for data driven manufacturing paradigms like Smart Manufacturing (a.k.a. Industry 4.0) serve as the technological backbone from which sustainable approaches to manufacturing can be implemented. Unfortunately, these technologies are typically associated with broader and deeper factory automation that is often too expensive and complex for the small and medium sized manufacturers (SMMs) that comprise the majority of manufacturing business in the USA and for whom their most valuable asset are the people whose jobs automation while replace. This paper describes an edge intelligent platform to integrate internet‐of‐things technologies with computing hardware, software, computational workflows for machine learning, and data ingestion, enabling SMMs to transition into smart manufacturing paradigms by leveraging the intelligence of their people. The platform leverages consumer grade electronics and sensors (affordable and portable), customized software with open source software packages (accessible), and existing communication network infrastructures (scalable). The software systems are implemented via Kubernetes orchestration of Docker containerization to ensure scalability and programmability. The platform is adaptive via computational workflow engines that produce information from data by processing with low‐cost edge computing devices while efficiently accessing resources of cloud servers as needed. The proposed edge platform connects workers to technological resources that provide computational intelligence (i.e., silicon‐based sensing and computation for data collection and contextualization) to enable decision making at the edge of advanced manufacturing.

Columnar formats for schemaless LSM-based document stores

In the last decade, document store database systems have gained more traction for storing and querying large volumes of semi-structured data. However, the flexibility of the document stores' data models has limited their ability to store data in a column-major layout --- making them less performant for analytical workloads than column store relational databases. In this paper, we propose several techniques based on piggy-backing on Log-Structured Merge (LSM) tree events and tailored to document stores to store data in a columnar layout. We first extend the Dremel format, a popular on-disk columnar format for semi-structured data, to comply with document stores' flexible data model. We then introduce a new columnar layout for organizing and storing data in LSM-based storage. We also highlight the potential of using query compilation techniques for document stores, where values' types are known only at runtime. We have implemented and evaluated our techniques to measure their impact on storage, data ingestion, and query performance in Apache AsterixDB. Our experiments show significant performance gains, improving the query execution time by orders of magnitude while minimally impacting ingestion performance.

Data Ingestion for AI in Prostate Cancer.

Prostate cancer (PCa) is one of the most prevalent cancers in the male population. Current clinical practices lead to overdiagnosis and overtreatment necessitating more effective tools for improving diagnosis, thus the quality of life of patients. Recent advances in infrastructure, computing power and artificial intelligence enable the collection of tremendous amounts of clinical and imaging data that could assist towards this end. ProCAncer-I project aims to develop an AI platform integrating imaging data and models and hosting the largest collection of PCa (mp)MRI, anonymized image data worldwide. In this paper, we present an overview of the overall architecture focusing on the data ingestion part of the platform. We describe the workflow followed for uploading the data and the main repositories for storing imaging data, clinical data and their corresponding metadata.

Workload-optimized sensor data store for industrial IoT gateways

In industrial Internet of Things (IoT) environments, sensor devices continue to generate a stream of sensor data, and the management of an ever-growing amount of data is a vital feature for IoT gateways. However, our preliminary analysis of some popular key–value stores used in IoT gateways revealed that none of the systems exploits the distinctive characteristics of IoT workloads – append-only and immutable – thus the systems show limitations in managing sensor data. To address this issue, in this study, we propose Indexing-of-Indexes (IoI) and LogFlush-and-Append (LFA) to exploit such characteristics. IoI is an indexing and data organization scheme designed to eliminate the notorious compaction-induced write amplification observed in legacy key–value stores, and LFA is a data ingestion scheme intended to remove double data write issues in legacy write-ahead logging implementations. We implement a prototype key–value store, Sen-Store, which incorporates our proposals, and we evaluate its performance using synthetic workloads and the TPCx-IoT benchmark. The evaluation results show that Sen-Store achieves up to 17.6× and 2.1× higher IoTps than industry-leading RocksDB and state-of-the-art IoTDB, respectively.

Black Swan Events and Intelligent Automation for Routine Safety Surveillance.

Effective identification of previously implausible safety signals is a core component of successful pharmacovigilance. Timely, reliable, and efficient data ingestion and related processing are critical to this. The term ‘black swan events’ was coined by Taleb to describe events with three attributes: unpredictability, severe and widespread consequences, and retrospective bias. These rare events are not well understood at their emergence but are often rationalized in retrospect as predictable. Pharmacovigilance strives to rapidly respond to potential black swan events associated with medicine or vaccine use. Machine learning (ML) is increasingly being explored in data ingestion tasks. In contrast to rule-based automation approaches, ML can use historical data (i.e., ‘training data’) to effectively predict emerging data patterns and support effective data intake, processing, and organisation. At first sight, this reliance on previous data might be considered a limitation when building ML models for effective data ingestion in systems that look to focus on the identification of potential black swan events. We argue that, first, some apparent black swan events—although unexpected medically—will exhibit data attributes similar to those of other safety data and not prove algorithmically unpredictable, and, second, standard and emerging ML approaches can still be robust to such data outliers with proper awareness and consideration in ML system design and with the incorporation of specific mitigatory and support strategies. We argue that effective approaches to managing data on potential black swan events are essential for trust and outline several strategies to address data on potential black swan events during data ingestion.

CP-BDHCA: Blockchain-Based Confidentiality-Privacy Preserving Big Data Scheme for Healthcare Clouds and Applications.

Healthcare big data (HBD) allows medical stakeholders to analyze, access, retrieve personal and electronic health records (EHR) of patients. Mostly, the records are stored on healthcare cloud and application (HCA) servers, and thus, are subjected to end-user latency, extensive computations, single-point failures, and security and privacy risks. A joint solution is required to address the issues of responsive analytics, coupled with high data ingestion in HBD and secure EHR access. Motivated from the research gaps, the paper proposes a scheme, that integrates blockchain (BC)-based confidentiality-privacy (CP) preserving scheme, CP-BDHCA, that operates in two phases. In the first phase, elliptic curve cryptographic (ECC)-based digital signature framework, HCA-ECC is proposed to establish a session key for secure communication among different healthcare entities. Then, in the second phase, a two-step authentication framework is proposed that integrates Rivest-Shamir-Adleman (RSA) and advanced encryption standard (AES), named as HCA-RSAE that safeguards the ecosystem against possible attack vectors. CP-BDAHCA is compared against existing HCA cloud applications in terms of parameters like response time, average delay, transaction and signing costs, signing and verifying of mined blocks, and resistance to DoS and DDoS attacks. We consider 10 BC nodes and create a real-world customized dataset to be used with SEER dataset. The dataset has 30,000 patient profiles, with 1000 clinical accounts. Based on the combined dataset the proposed scheme outperforms traditional schemes like AI4SAFE, TEE, Secret, and IIoTEED, with a lower response time. For example, the scheme has a very less response time of 300 ms in DDoS. The average signing cost of mined BC transactions is 3,34 seconds, and for 205 transactions, has a signing delay of 1405 ms, with improved accuracy of ≈ 12% than conventional state-of-the-art approaches.

Building Resilient Data Ingestion Pipelines for Third-Party Vendor Data Integration

In this report, the author provides a review of the design and operation of the resilient data ingestion architecture with a particular emphasis on the issues associated with third-party data vendor integration. As more and more companies have made information a principal factor in their business strategies, it has become equally imperative to obtain external data in a coherent manner. This paper describes ways of handling Enhanced Data Ingestion Reliability Efficiency and Adaptability and other challenges such as Data quality Error Handling and Scalability.

SciKit Digital Health: Python Package for Streamlined Wearable Inertial Sensor Data Processing.

Wearable inertial sensors are providing enhanced insight into patient mobility and health. Significant research efforts have focused on wearable algorithm design and deployment in both research and clinical settings; however, open-source, general-purpose software tools for processing various activities of daily living are relatively scarce. Furthermore, few studies include code for replication or off-the-shelf software packages. In this work, we introduce SciKit Digital Health (SKDH), a Python software package (Python Software Foundation) containing various algorithms for deriving clinical features of gait, sit to stand, physical activity, and sleep, wrapped in an easily extensible framework. SKDH combines data ingestion, preprocessing, and data analysis methods geared toward modern data science workflows and streamlines the generation of digital endpoints in “good practice” environments by combining all the necessary data processing steps in a single pipeline. Our package simplifies the construction of new data processing pipelines and promotes reproducibility by following a convention over configuration approach, standardizing most settings on physiologically reasonable defaults in healthy adult populations or those with mild impairment. SKDH is open source, as well as free to use and extend under a permissive Massachusetts Institute of Technology license, and is available from GitHub (PfizerRD/scikit-digital-health), the Python Package Index, and the conda-forge channel of Anaconda.